Description
Definition at line 56 of file DocumentMarkup.h.
Inheritance diagram for Sawyer::Document::Markup::TokenStream:
Collaboration diagram for Sawyer::Document::Markup::TokenStream:
Public Member Functions | |
| TokenStream (const std::string &s) | |
| Token | scanNextToken (const Container::LineVector &content, size_t &at) |
| Function that obtains the next token. More... | |
 Public Member Functions inherited from Sawyer::Lexer::TokenStream< Token > | |
| TokenStream (const boost::filesystem::path &fileName) | |
| Create a token stream from the contents of a file. | |
| TokenStream (const std::string &inputString) | |
| Create a token stream from a string. More... | |
| TokenStream (const Container::Buffer< size_t, char >::Ptr &buffer) | |
| Create a token stream from a buffer. More... | |
| const std::string & | name () const |
| Property: Name of stream. | |
| const Token & | current () |
| Return the current token. More... | |
| bool | atEof () |
| Returns true if the stream is at the end. More... | |
| const Token & | operator[] (size_t lookahead) |
| Return the current or future token. More... | |
| void | consume (size_t n=1) |
| Consume some tokens. More... | |
| std::pair< size_t, size_t > | location (size_t position) |
| Return the line number and offset for an input position. More... | |
| std::pair< size_t, size_t > | locationEof () |
| Returns the last line index and character offset. | |
| std::string | lineString (size_t lineIdx) |
| Return the entire string for some line index. | |
| std::string | lexeme (const Token &t) |
| Return the lexeme for a token. More... | |
| std::string | lexeme () |
| Return the lexeme for a token. More... | |
| bool | isa (const Token &t, typename Token::TokenEnum type) |
| Determine whether token is a specific type. More... | |
| bool | isa (typename Token::TokenEnum type) |
| Determine whether token is a specific type. More... | |
| bool | match (const Token &t, const char *s) |
| Determine whether a token matches a string. More... | |
| bool | match (const char *s) |
| Determine whether a token matches a string. More... | |
Additional Inherited Members | |
| Public Types inherited from Sawyer::Lexer::TokenStream< Token > | |
| typedef Token | Token |
Member Function Documentation
◆ scanNextToken()
|
virtual |
Function that obtains the next token.
Subclasses implement this function to obtain the next token that starts at or after the specified input position. Upon return, the function should adjust at to point to the next position for scanning a token, which is usually the first character after the returned token's lexeme. If the scanner reaches the end of input or any condition that it deems to be the end then it should return the EOF token (a default-constructed token), after which this function will not be called again.
Implements Sawyer::Lexer::TokenStream< Token >.
The documentation for this class was generated from the following file:
