[10] token = identifier
| keyword
| boolean
| number
| character
| string
| named-constant
| glyph-identifier
| (
| )
| '
| .
| &grave.
| ,
| ,@
[11] delimiter = whitespace
| (
| )
| "
| ;
[12] whitespace = space
| record-start
| record-end
| tab
| form-feed
| separator-character
[13] comment = ; any-character-except-record-end*
[14] atmosphere = whitespace
| comment
[15] intertoken-space = atmosphere*
Whitespace characters are spaces, record starts, record ends, and
separator-characters.
Whitespace is used for improved readability and,
as necessary, to separate tokens from each other, a token being an
indivisible lexical unit such as an identifier or number, but is
otherwise insignificant. Whitespace may occur between any two tokens,
but not within a token. Whitespace may also occur inside a string,
where it is significant.
A semicolon (;) indicates the start of a
comment. The comment continues to the
end of the record on which the semicolon appears. Comments are invisible,
but the record end is visible as whitespace. This
prevents a comment from appearing in the middle of an identifier or
number.
intertoken-space may occur on either side of any token, but not
within a token.
Tokens which require implicit termination (identifiers, numbers,
characters, dot, and #! constants) may be terminated by any
delimiter, but not necessarily by anything else.