Skip to content

Commit 420f4d3

Browse files
committed
Add a lexer rule for punctuation
This is intended to help define what a "token" is via the grammar (and to fill a missing hole in our token definition). I waffled on how to define delimiters, whether they should be separate somehow. In practice I think it should be fine to clump them all together. This mainly only matters for TokenTree which already excludes the delimiters.
1 parent 1c65870 commit 420f4d3

File tree

1 file changed

+58
-0
lines changed

1 file changed

+58
-0
lines changed

src/tokens.md

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -777,6 +777,64 @@ r[lex.token.life.raw.edition2021]
777777
r[lex.token.punct]
778778
## Punctuation
779779

780+
r[lex.token.punct.syntax]
781+
> **<sup>Lexer</sup>**\
782+
> Token :\
783+
> PUNCTUATION :\
784+
> &nbsp;&nbsp; &nbsp;&nbsp; `=`\
785+
> &nbsp;&nbsp; | `<`\
786+
> &nbsp;&nbsp; | `<=`\
787+
> &nbsp;&nbsp; | `==`\
788+
> &nbsp;&nbsp; | `!=`\
789+
> &nbsp;&nbsp; | `>=`\
790+
> &nbsp;&nbsp; | `>`\
791+
> &nbsp;&nbsp; | `&&`\
792+
> &nbsp;&nbsp; | `||`\
793+
> &nbsp;&nbsp; | `!`\
794+
> &nbsp;&nbsp; | `~`\
795+
> &nbsp;&nbsp; | `+`\
796+
> &nbsp;&nbsp; | `-`\
797+
> &nbsp;&nbsp; | `*`\
798+
> &nbsp;&nbsp; | `/`\
799+
> &nbsp;&nbsp; | `%`\
800+
> &nbsp;&nbsp; | `^`\
801+
> &nbsp;&nbsp; | `&`\
802+
> &nbsp;&nbsp; | `|`\
803+
> &nbsp;&nbsp; | `<<`\
804+
> &nbsp;&nbsp; | `>>`\
805+
> &nbsp;&nbsp; | `+=`\
806+
> &nbsp;&nbsp; | `-=`\
807+
> &nbsp;&nbsp; | `*=`\
808+
> &nbsp;&nbsp; | `/=`\
809+
> &nbsp;&nbsp; | `%=`\
810+
> &nbsp;&nbsp; | `^=`\
811+
> &nbsp;&nbsp; | `&=`\
812+
> &nbsp;&nbsp; | `|=`\
813+
> &nbsp;&nbsp; | `<<=`\
814+
> &nbsp;&nbsp; | `>>=`\
815+
> &nbsp;&nbsp; | `@`\
816+
> &nbsp;&nbsp; | `.`\
817+
> &nbsp;&nbsp; | `..`\
818+
> &nbsp;&nbsp; | `...`\
819+
> &nbsp;&nbsp; | `..=`\
820+
> &nbsp;&nbsp; | `,`\
821+
> &nbsp;&nbsp; | `;`\
822+
> &nbsp;&nbsp; | `:`\
823+
> &nbsp;&nbsp; | `::`\
824+
> &nbsp;&nbsp; | `->`\
825+
> &nbsp;&nbsp; | `<-`\
826+
> &nbsp;&nbsp; | `=>`\
827+
> &nbsp;&nbsp; | `#`\
828+
> &nbsp;&nbsp; | `$`\
829+
> &nbsp;&nbsp; | `?`\
830+
> &nbsp;&nbsp; | `_`\
831+
> &nbsp;&nbsp; | `{`\
832+
> &nbsp;&nbsp; | `}`\
833+
> &nbsp;&nbsp; | `[`\
834+
> &nbsp;&nbsp; | `]`\
835+
> &nbsp;&nbsp; | `(`\
836+
> &nbsp;&nbsp; | `)`
837+
780838
r[lex.token.punct.intro]
781839
Punctuation symbol tokens are listed here for completeness. Their individual
782840
usages and meanings are defined in the linked pages.

0 commit comments

Comments
 (0)