Translate a raw string into tokens for the parser. To implement
an extension to the lexer, one may process its result before
giving it to the parser. To implement an extension to the
parser, one may extend it using the constructor Tag
from type tok
and/or using the extensions mechanism
of the parser (cf. the optional argument extensions
).
The main difference is that Tag
is processed by the parser
in highest priority whereas functions in extensions
are applied
with lowest priority.
string_of_tokens t
return the string corresponding to the token
list t
.
length t
number of characters of the string represented as t
(i.e. String.length(string_of_token t)
).
string_of_token tk
return the string corresponding to the token
tk
.