3.1. 構文¶

この章では、プログラムテキストの構文解析がどのように行われるかを述べる。

3.1.1. 解析の手順¶

プログラムテキスト は unicode scalar value の列である。

プログラムテキストは、まず形態素の列に分割される。分割は、最長一致法をもちいて、先頭から末尾まで行われる。

次に形態素の列は、トークンでない形態素を省くことにより、トークンの列に変換される。

ついで、トークンの列は、 program の抽象構文木としてパースされる。

3.1.2. 形態素¶

3.1.2.1. Whitespace and comments¶

正規表現	形態素の種別
`\u0020\|\u000a\|\u000d\u000a`	Whitespace
`#[^\u000d\u000a]*`	Comment

Whitespace is used to make a program text easy to read and to separate tokens from each other. Space (U+0020), LF (U+000a), and CR+LF (U+000d, U+000a) are whitespace character sequences.

注釈

Horizontal tab (U+0009) is not a valid whitespace character. This design decision is intended to avoid bikeshed discussion about indentation.

The categories of morphemes (, [ and { may differ when they are located after whitespace characters. See the description of token types for detail.

Comments are used to describe the program text. A number sign # (U+0023) indicates a start of a comment. The comment continues till the end of the line, which is before the CR or LF character or the end of the program text. Comments are treated as whitespace characters.

stdout.print_line('foo'*2)      # => foofoo

stdout.print_line( 'foo' * 2 )  # => foofoo

# Comment line
do_something  # trailing comment

Whitespace characters and comments are not tokens.

3.1.2.2. Symbol tokens¶

正規表現	形態素の種別
`[a-z_][a-z0-9_?]*`	Fun symbol token
`([a-z_][a-z0-9_?])?[A-Z][a-zA-Z0-9_?]`	Data symbol token

A symbol token consists of a leading ascii letter (a-zA-Z) or an underscore _ (U+005f), and a trailing sequence of zero or more ascii letters (a-zA-Z), ascii digits (0-9), underscores _ (U+005f) and question marks ? (U+0x3f).

There are two types of symbol tokens: function symbol tokens and data symbol tokens.

A function symbol token is a symbol token which represents a function symbol. Function symbols are commonly used for names of variables which contain functions. The following are exmaples of function symbols.

any?
_loop
take_5

A data symbol token is a symbol token which represents a data symbol. Data symbols are commonly used for names of variables which contain ordinary values. The following are examples of data symbols.

More_lines?
ArrayList_class
FLAT_MAP
_HASH_TABLE
rarely_Used

3.1.2.3. Num tokens¶

正規表現	形態素の種別
`[0-9][0-9_](\.[0-9][0-9_])?(?![a-zA-Z0-9_?])`	Num token (base10)
`0b_[01][01_](?![a-zA-Z0-9_?])`	Num token (base2)
`0x_[0-9a-f][0-9a-f_](?![a-zA-Z0-9_?])`	Num token (base16)

There are three types of num tokens: base10, base16 and base2. A base10 num token represents a num in decimal, base16 in hexadecimal, and base2 in binary.

The fractional part can be represented only by base10 num tokens. Digits before the period . (U+002e) represents the integer portion, and digits after the period represents the fractional portion.

Underscore characters _ (U+005f) can be placed for spacing.

These are examples of num tokens, each of which represents 42.

42
42__
0042
0x2a
0b_10_1010

These are examples of num tokens with the fractional portion.

0.0
0.001
3.141_592_653

All base10, base2 and base16 num tokens cannot be directly followed by a character which can form a symbol. This limitation is represented as (?![a-zA-Z0-9_?]). Thus, for example, a code fragment 24h causes a syntax error.

注釈

Without this rule, for example, 0b123 is parsed as 0b1 and 23. It is certainly error prone.

Each num token represents a number the scale of which is the count of digits of the fractional portion, and the mantissa of which is the integer made of the digits.

3.1.2.4. 文字列トークン¶

正規表現	形態素の種別
`'(''\|[^'\u000d\000a])*'`	文字列トークン（単純）
`"([^"\u000d\u000a\\]\|\\[0abtnvfre"\\]\|\\x\{[0-9a-f]{1,6}\})*"`	文字列トークン（リッチ）

文字列トークン には単純とリッチの二種類がある。

単純文字列トークン の中では、ふたつの一重引用符 ' (U+0027)にはさまれた文字が、文字列の中身になる。ただし、一重引用符自体を文字列に含みたい場合は、一重引用符をふたつ続ける。

単純文字列トークンの例:

'Hello world'
'Let''s go!' (これは "Let's go!" を表す)

リッチ文字列トークン の中では、ふたつの二重引用符 " (U+0022)にはさまれた文字が、文字列の中身になる。このトークンの中では、バックスラッシュ \ (U+005c)で始まる列が特殊な文字を表す。たとえば、ラインフィード (\n)や二重引用符(\")のように。

リッチ文字列トークンの例:

"Let's go!"
"GET /index.html HTTP/1.1\r\nHost: host.example.org\r\n"

バックスラッシュ記法 の一覧:

記法	Unicode	説明
`\0`	U+0000	ヌル文字
`\a`	U+0007	ベル文字
`\b`	U+0008	後退
`\t`	U+0009	水平タブ
`\n`	U+000a	ラインフィード
`\v`	U+000b	垂直タブ
`\f`	U+000c	書式送り
`\r`	U+000d	キャリッジリターン
`\e`	U+001b	エスケープ文字
`\"`	U+0022	二重引用符 `"`
`\\`	U+005c	バックスラッシュ `\`
`\x{xxxxxx}`	U+xxxxxx	Unicode scalar value で指定された文字。xxxxxxは1つから6つ並んだ16進数の桁(0-9a-f)で、0からd7ff₁₆ またはe000₁₆ から10ffff₁₆ の範囲（いずれも包含的）の整数を表す。

注釈

正規表現が示すように、単純文字列トークン、リッチ文字列トークンともに、複数行のプログラム行にまたがることはできない。これは、エディタープラグインなどの外部ツールが、容易に字句解析できるようにするためである。複数行に渡るテキストをプログラムから扱いたい場合は、リッチ文字列トークンでバックスラッシュ記法を使うか、データファイルモジュールを同梱すると良い。

3.1.2.5. Binding token¶

正規表現	形態素の種別
`\binding(?![a-zA-Z0-9_?])`	Binding token

A binding token is literally \binding. It is used for a binding expression.

3.1.2.6. Punctuation marks and operators¶

Tokens other than described above are punctuation marks or operators.

Some morphemes, like $, are reduced to different tokens depending on whether it is placed just after a whitespace character or not. If the morpheme is placed after a whitespace character, the token is represented with a prefix ws, like ws$. If the morpheme is not placed after a whitespace character, the token is represented with a prefix nows, like nows$.

Other morphemes, like ==, are reduced to a single token not depending on whether it is placed just after a whitespace character or not. Those tokens are represented literally like ==.

The list of punctuation marks and operators:

Token	Note
`!`	`op_lognot` operator
`~`	`op_not` operator
`<-`	`op_store` operator
`\|\|`	`op_logor` operator
`&&`	`op_logand` operator
`==`	`op_eq` operator
`!=`	not-equal operator
`<`	less-than operator
`>`	greater-than operator
`<=`	`op_le` operator
`>=`	greater-than-or-equal-to operator
`\|`	`op_or` operator
`^`	`op_xor` operator
`&`	`op_and` operator
`<<`	`op_shl` operator
`>>`	`op_shr` operator
`+`	`op_add` operator
`-`	`op_sub` operator, or `op_minus` operator
`*`	`op_mul` operator
`/`	`op_div` operator
`//`	`op_intdiv` operator
`%`	`op_rem` operator
`=`	Let clauses
`nows:`	Local variable references
`ws:`	Member variable references
`nows$`	Local variable load of a function symbol
`ws$`	Member variable load of a function symbol
`.`	Access to variables or functions
`...`	Elements spreading
`nows[`	Vectors, formal receivers, or actual receivers
`ws[`	Vectors
`]`	Closing
`nows{`	Function expressions or function arguments
`ws{`	Function expressions
`}`	Closing
`nows(`	Parentheses expressions, formal arguments, or actual arguments
`ws(`	Parentheses expressions
`)`	Closing

3.1.3. Abstract source tree¶

The list describes the parsing rule of the program abstract source tree, or AST, from the token sequence of a program text.

There are several shift/reduce conflicts in the rules, and the parser always choose to shift.

program           ::=  toplevel
toplevel          ::=  empty
                       expression toplevel
seq               ::=  empty
                       substantial_seq
substantial_seq   ::=  expression
                       expression substantial_seq
                       expression '=' expression substantial_seq
expression        ::=  store_op
store_op          ::=  logor_op
                       logor_op '<-' logor_op
logor_op          ::=  logand_op
                       logand_op '||' logor_op
logand_op         ::=  relation_op
                       relation_op '&&' logand_op
relation_op       ::=  add_op
                       add_op '==' add_op
                       add_op '!=' add_op
                       add_op '<' add_op
                       add_op '>' add_op
                       add_op '<=' add_op
                       add_op '>=' add_op
add_op            ::=  multiply_op
                       add_op '+' multiply_op
                       add_op '-' multiply_op
                       add_op '|' multiply_op
                       add_op '^' multiply_op
multiply_op       ::=  unary_op
                       multiply_op '*' unary_op
                       multiply_op '/' unary_op
                       multiply_op '//' unary_op
                       multiply_op '%' unary_op
                       multiply_op '&' unary_op
                       multiply_op '<<' unary_op
                       multiply_op '>>' unary_op
unary_op          ::=  primary
                       '-' unary_op
                       '!' unary_op
                       '~' unary_op
primary           ::=  num
                       str
                       binding
                       paren
                       vec
                       fun
                       local_load
                       member_load
                       local_varref
                       member_varref
                       local_call
                       member_call
num               ::=  NUM
str               ::=  STRING
binding           ::=  BINDING
paren             ::=  'nows(' seq ')'
                       'ws(' seq ')'
vec               ::=  'nows[' vec_body ']'
                       'ws[' vec_body ']'
fun               ::=  'nows{' fun_body '}'
                       'ws{' fun_body '}'
local_load        ::=  DATA_SYM | nows'$' FUN_SYM | ws'$' FUN_SYM
local_varref      ::=  'nows:' DATA_SYM
                       'ws:' DATA_SYM
                       'nows:' FUN_SYM
                       'ws:' FUN_SYM
local_call        ::=  FUN_SYM recv args
member_load       ::=  primary '.' DATA_SYM
                       primary 'nows$' FUN_SYM
member_varref     ::=  primary 'nows:' DATA_SYM
                       primary 'nows:' FUN_SYM
member_call       ::=  primary '.' FUN_SYM recv args
recv              ::=  empty
                       'nows[' expression ']'
args              ::=  paren_args fun_args
paren_args        ::=  empty
                       'nows(' vec_body ')'
fun_args          ::=  empty
                       fun_arg fun_args
fun_arg           ::=  'nows{' fun_body '}'
vec_body          ::=  empty
                       elements_producer vec_body
elements_producer ::=  expression
                       '...' expression
fun_body          ::=  formal_receiver formal_args seq
formal_receiver   ::=  empty
                       'nows[' expression ']'
formal_args       ::=  empty
                       'nows(' vec_body ')'
empty             ::=

The following identifiers represent token types.

Token type	説明
FUN_SYM	Function symbol token such as `print`
DATA_SYM	Data symbol token such as `Count`
NUM	Num token such as `42` or `3.14159`
STRING	String token such as `'Peng!'`
BINDING	Binding token `\binding`
Those enclosed by single quotation marks	Punctuation marks or operators