3.4 Building and using the Lexer

To build the lexer, the function lex.lex() is used. This function uses Python reflection (or introspection) to read the the regular expression rules out of the calling context and build the lexer. Once the lexer has been built, two functions can be used to control the lexer.

• lex.input(data). Reset the lexer and store a new input string.
• lex.token(). Return the next token. Returns a special LexToken instance on success or None if the end of the input text has been reached.

If desired, the lexer can also be used as an object. The lex() returns a Lexer object that can be used for this purpose. For example:

lexer = lex.lex()
lexer.input(sometext)
while 1:
    tok = lexer.token()
    if not tok: break
    print tok

This latter technique should be used if you intend to use multiple lexers in your application. Simply define each lexer in its own module and use the object returned by lex() as appropriate.
Note: The global functions lex.input() and lex.token() are bound to the input() and token() methods of the last lexer created by the lex module.

Example: