ROSE  0.11.96.11
Public Types | Public Member Functions | List of all members
Sawyer::Lexer::Token< T > Class Template Reference

Description

template<typename T>
class Sawyer::Lexer::Token< T >

Represents one token of input.

Each token has a user-defined type which is some enumerated constant, or no type at all which means the token represents the end of the input stream. Tokens do not store their own lexeme, but rather point to the beginning and end of their lexeme in the input stream.

Definition at line 28 of file Lexer.h.

#include <Lexer.h>

Public Types

typedef T TokenEnum
 

Public Member Functions

 Token ()
 Construct an EOF token.
 
 Token (TokenEnum type, size_t begin, size_t end)
 Construct a token. More...
 
bool isEof () const
 Whether this is an EOF token. More...
 
TokenEnum type () const
 Returns the token. More...
 
size_t begin () const
 Token lexeme starting position. More...
 
size_t end () const
 Token lexeme ending position. More...
 

Constructor & Destructor Documentation

◆ Token()

template<typename T >
Sawyer::Lexer::Token< T >::Token ( TokenEnum  type,
size_t  begin,
size_t  end 
)
inline

Construct a token.

The token has the specified type and its lexeme are the characters at the specified position in the input stream. The begin points to the first character of the lexeme and end points to one past the last character. A token's lexeme is allowed to be the empty string by specifying the same value for begin and end, however, end must never be less than begin.

Definition at line 47 of file Lexer.h.

Member Function Documentation

◆ isEof()

template<typename T >
bool Sawyer::Lexer::Token< T >::isEof ( ) const
inline

Whether this is an EOF token.

An EOF token is a special token that has no type and an empty lexeme. EOF tokens are constructed by the default constructor.

Definition at line 56 of file Lexer.h.

◆ type()

template<typename T >
TokenEnum Sawyer::Lexer::Token< T >::type ( ) const
inline

Returns the token.

Since EOF tokens have no type, this must not be called for an EOF token.

Definition at line 65 of file Lexer.h.

◆ begin()

template<typename T >
size_t Sawyer::Lexer::Token< T >::begin ( ) const
inline

Token lexeme starting position.

This is the starting offset in the input for the first character of this token's lexeme.

Definition at line 72 of file Lexer.h.

◆ end()

template<typename T >
size_t Sawyer::Lexer::Token< T >::end ( ) const
inline

Token lexeme ending position.

This is the offset in the input for one position past the last character of this token's lexeme. It is guaranteed to be greater than or equal to the begin position. EOF tokens will always have a begin equal to the end, but other empty non-EOF tokens are also possible.

Definition at line 81 of file Lexer.h.


The documentation for this class was generated from the following file: