Scanners

Provides the scanners for the tokenizer.

The role of a scanner is to recognize a sequence of characters and to produce a ScanResult.

All the UTF-8 characters that Norg specifies as a whitespace.

All the UTF-8 characters that are punctuation in Norg specification. See the norg specification

All the registered Kinds.Kind that Scanners.scan will try when consuming entries.

Stores the result of a scanning operation. If length is 0 that means that no matching token was found.

scan(pattern, input)

Scan the given input for the given pattern.

It will produce a Scanners.ScanResult.

If pattern is given, then try to fit the given patter at the start of input. If pattern is :

• a ScanStrategy subtype : scan with the given strategy (e.g. Word or Whitespace)
• a Kind : parse for the given kind.
• an AbstractString : input must startswith pattern.
• an AbstractArray : call scan on each element of pattern until one matches.
• a Set{Char} : first character must be included in pattern.

scan(input; line=0, charnum=0)

Scan the given input for Scanners.TOKENKIND_PARSING_ORDER until one returns a successful Scanners.ScanResult or throw an error if none succeed.

This will return a Tokens.Token.