#include <deque>#include <istream>
Include dependency graph for token.hpp:
This graph shows which files directly or indirectly include this file:
Go to the source code of this file.
Classes | |
| class | Token |
| Value class representing a token. More... | |
Typedefs | |
| typedef std::deque< Token > | TokenSequenceType |
Functions | |
| TokenSequenceType | tokenize (std::istream &seq) |
| Split a stream into a sequnce of tokens. More... | |
Detailed Description
Defines the Token and TokenSequence types, and associated functions.
Typedef Documentation
◆ TokenSequenceType
Define the token sequence using a std container. Any supporting sequential access should do.
Function Documentation
◆ tokenize()
| TokenSequenceType tokenize | ( | std::istream & | seq | ) |
Split a stream into a sequnce of tokens.
- Parameters
-
seq the input character stream
- Returns
- The sequence of tokens
Split a stream into a sequnce of tokens where a token is one of OPEN or CLOSE or any space-delimited string
Ignores any whitespace and comments (from any ";" to end-of-line).