Skip to content

Commit a5a616f

Browse files
authored
Adds support for string literals and identifiers in Scanner. (#15)
Refactors the PEG to make each keyword and explicit rule that has the form `KeywordName_`. This allows us to easily refer to them in rules without having to have magic strings. The rationale for the `_` suffix is to make sure it doesn't collide with the other grammar rules and not have a wordy suffix like `Keyword` on every rule. Also: * Adds case folding to keywords. * Makes error cases in the scanner tests less brittle. * Adds `Token` to the `Query` rule and updates tests for `recognize_partiql`.
1 parent 2777229 commit a5a616f

File tree

4 files changed

+773
-285
lines changed

4 files changed

+773
-285
lines changed

partiql-parser/src/lib.rs

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,12 +15,13 @@
1515
//! fn main() -> ParserResult<()> {
1616
//! use partiql_parser::scanner::Content::*;
1717
//!
18-
//! let mut scanner = scanner("SELECT FROM");
18+
//! let mut scanner = scanner("SELECT '🦄💩'");
1919
//! let first = scanner.next_token()?;
2020
//!
2121
//! // get the parsed variant of the token
2222
//! match first.content() {
2323
//! Keyword(kw) => assert_eq!("SELECT", kw),
24+
//! Identifier(_) | StringLiteral(_) => panic!("Didn't get a keyword!"),
2425
//! }
2526
//! // the entire text of a token can be fetched--which looks the roughly the
2627
//! // same for a keyword.
@@ -29,7 +30,8 @@
2930
//! let second = scanner.next_token()?;
3031
//! // get the parsed variant of the token
3132
//! match second.content() {
32-
//! Keyword(kw) => assert_eq!("FROM", kw),
33+
//! StringLiteral(text) => assert_eq!("🦄💩", text),
34+
//! Keyword(_) | Identifier(_) => panic!("Didn't get a string literal!"),
3335
//! }
3436
//! // the other thing we can do is get line/column information from a token
3537
//! assert_eq!(LineAndColumn::try_at(1, 8)?, second.start());

0 commit comments

Comments
 (0)