30 Token(
const std::string & str);
TokenType
a public enum defining the possible token types.
Definition: token.hpp:21
std::string asString() const
return the token rendered as a string
Definition: token.cpp:20
Token(TokenType t)
construct a token of type t (if string default to empty value)
Definition: token.cpp:12
Value class representing a token.
Definition: token.hpp:15
TokenSequenceType tokenize(std::istream &seq)
Split a stream into a sequnce of tokens.
Definition: token.cpp:40
std::deque< Token > TokenSequenceType
Definition: token.hpp:47
TokenType type() const
return the type of the token
Definition: token.cpp:16