ROSE
0.11.96.11
|
Represents one token of input.
Each token has a user-defined type which is some enumerated constant, or no type at all which means the token represents the end of the input stream. Tokens do not store their own lexeme, but rather point to the beginning and end of their lexeme in the input stream.
#include <Lexer.h>
Public Types | |
typedef T | TokenEnum |
Public Member Functions | |
Token () | |
Construct an EOF token. | |
Token (TokenEnum type, size_t begin, size_t end) | |
Construct a token. More... | |
bool | isEof () const |
Whether this is an EOF token. More... | |
TokenEnum | type () const |
Returns the token. More... | |
size_t | begin () const |
Token lexeme starting position. More... | |
size_t | end () const |
Token lexeme ending position. More... | |
|
inline |
Construct a token.
The token has the specified type and its lexeme are the characters at the specified position in the input stream. The begin
points to the first character of the lexeme and end
points to one past the last character. A token's lexeme is allowed to be the empty string by specifying the same value for begin
and end
, however, end
must never be less than begin
.
|
inline |
|
inline |
|
inline |
|
inline |