4.3.2 Special Tokens

In addition to the user defined tokens, Bison generates a few special tokens that yylex may return.

The YYEOF token denotes the end of file, and signals to the parser that there is nothing left afterwards. See Calling Convention for yylex, for an example.

Returning YYUNDEF tells the parser that some lexical error was found. It will emit an error message about an “invalid token”, and enter error-recovery (see Error Recovery). Returning an unknown token kind results in the exact same behavior.

Returning YYerror requires the parser to enter error-recovery without emitting an error message. This way the lexical analyzer can produce an accurate error messages about the invalid input (something the parser cannot do), and yet benefit from the error-recovery features of the parser.

int
yylex (void)
{
  …
  switch (c)
    {
      …
      case '0': case '1': case '2': case '3': case '4':
      case '5': case '6': case '7': case '8': case '9':
        …
        return TOK_NUM;
      …
      case EOF:
        return YYEOF;
      default:
        yyerror ("syntax error: invalid character: %c", c);
        return YYerror;
    }
}