ROSE  0.11.96.11
Public Member Functions | List of all members
Sawyer::Document::Markup::TokenStream Class Reference

Description

Definition at line 56 of file DocumentMarkup.h.

Inheritance diagram for Sawyer::Document::Markup::TokenStream:
Inheritance graph
[legend]
Collaboration diagram for Sawyer::Document::Markup::TokenStream:
Collaboration graph
[legend]

Public Member Functions

 TokenStream (const std::string &s)
 
Token scanNextToken (const Container::LineVector &content, size_t &at)
 Function that obtains the next token. More...
 
- Public Member Functions inherited from Sawyer::Lexer::TokenStream< Token >
 TokenStream (const boost::filesystem::path &fileName)
 Create a token stream from the contents of a file.
 
 TokenStream (const std::string &inputString)
 Create a token stream from a string. More...
 
 TokenStream (const Container::Buffer< size_t, char >::Ptr &buffer)
 Create a token stream from a buffer. More...
 
const std::string & name () const
 Property: Name of stream.
 
const Token & current ()
 Return the current token. More...
 
bool atEof ()
 Returns true if the stream is at the end. More...
 
const Token & operator[] (size_t lookahead)
 Return the current or future token. More...
 
void consume (size_t n=1)
 Consume some tokens. More...
 
std::pair< size_t, size_t > location (size_t position)
 Return the line number and offset for an input position. More...
 
std::pair< size_t, size_t > locationEof ()
 Returns the last line index and character offset.
 
std::string lineString (size_t lineIdx)
 Return the entire string for some line index.
 
std::string lexeme (const Token &t)
 Return the lexeme for a token. More...
 
std::string lexeme ()
 Return the lexeme for a token. More...
 
bool isa (const Token &t, typename Token::TokenEnum type)
 Determine whether token is a specific type. More...
 
bool isa (typename Token::TokenEnum type)
 Determine whether token is a specific type. More...
 
bool match (const Token &t, const char *s)
 Determine whether a token matches a string. More...
 
bool match (const char *s)
 Determine whether a token matches a string. More...
 

Additional Inherited Members

- Public Types inherited from Sawyer::Lexer::TokenStream< Token >
typedef Token Token
 

Member Function Documentation

◆ scanNextToken()

Token Sawyer::Document::Markup::TokenStream::scanNextToken ( const Container::LineVector content,
size_t &  at 
)
virtual

Function that obtains the next token.

Subclasses implement this function to obtain the next token that starts at or after the specified input position. Upon return, the function should adjust at to point to the next position for scanning a token, which is usually the first character after the returned token's lexeme. If the scanner reaches the end of input or any condition that it deems to be the end then it should return the EOF token (a default-constructed token), after which this function will not be called again.

Implements Sawyer::Lexer::TokenStream< Token >.


The documentation for this class was generated from the following file: