Represents a single scanned token in the data structure. More...
#include <ScannedTokens.hpp>
Public Attributes | |
| CodeSpan | codeSpan |
| The start and end positions of the token's text within the multi-line source file. More... | |
| std::string_view | text |
| UTF-8 formatted string of the token. More... | |
| TokenT | token |
| The token type for this scanned token. More... | |
Represents a single scanned token in the data structure.
When accessed via the scanner API, the scanned tokens returned from the get() method contain full code span information.
The supplied scanned tokens MUST end in an EndOfFile token.
If full code span information is required with random access, the higher overhead RandomAccessScannedTokens class may be used to wrap the ScannedTokens class.
| CodeSpan codeSpan |
The start and end positions of the token's text within the multi-line source file.
| std::string_view text |
UTF-8 formatted string of the token.
| TokenT token |
The token type for this scanned token.