Encapsulation of a set of language tokens, source text, and start offsets. More...
#include <ScannedTokens.hpp>
Public Member Functions | |
| std::string_view | getText (unsigned int index) |
| Get the text of the token with the specified index. More... | |
| std::shared_ptr< const Resource::Uri > | getUri () |
| Get the input uri. More... | |
| std::string && | moveTextOut () |
| Move the input text string to its final destination. More... | |
Encapsulation of a set of language tokens, source text, and start offsets.
This is the data structure returned by the scanner. It is structured to use the minimum amount of memory necessary in order to hold information on the whole scanned source file.
The supplied scanned tokens MUST end in an EndOfFile token.
There are three APIs defined by external adaptor classes:
When used by a parser, the scanner API provides a traditional scanner API with infinite lookahead and putback. The whitespace policy may be set dynamically by pushing and popping whitespace modes during consummation of the tokens.
For other uses, the random access and iterative APIs may be used.
The data structure is not random access. The token code span and string information is constructed serially in the scanner API. This permits the data structure to be as small as possible.
If random access is required, the RandomAccessScannedTokens wrapper may be built. Note that the memory overhead of the RandomAccessScannedTokens wrapper is far greater than that of the ScannedTokens data structure.
The scanned token type TokenT must be an enum containing at least the following entries:
In addition to the LineBreak token, the CommentLine enum entry is also considered to end with a line break.
| TokenT | the language token enum type |
|
inline |
Get the text of the token with the specified index.
|
inline |
Get the input uri.
|
inline |
Move the input text string to its final destination.
The scanned tokens instance must not be used after the move is performed.