Tokens

Provides the tokens for the tokenizer.

A Tokens.Token stores its value, a Kinds.Kind and a Tokens.TokenPosition

A Norg Token has a Kinds.kind (i.e. K"EndLine"), a position, and a value.

See also: Tokens.TokenPosition

 Token(kind, line, char, value)

Create a Token of kind kind with value value at line and char number char.

Stores the position of a token in the input file (line and char).

char(x)

Return the character number in the line corresponding to position or token x.

line(x)

Return the line number corresponding to the position or token x.

Produce Tokens.Token vectors from a string using tokenize.

tokenize(input)

Produce Tokens.Token vectors from an input string.