Separate parser crate

[alensiljak]

The idea here is to create crates for parser implementation(s). These would be used by the main application but could also be used by external projects.
Having parsers in separate crates, with all the dependencies, would make swapping different parser implementations much easier. This would be really helpful until the application is stabilized. Also, should other parsers appear (like Antlr officially supporting Rust), it would enable switching implementations.

It would be useful to compile parser(s) to WASM and use in JavaScript code.

Questions to explore:

• what is the exact expected output of the parser? AST, domain model
• what structure does the Ledger parser provide?

Resources:

• Parser Resources wiki page collects the useful resources mentioned in this discussion
• Modular Architecture #25
• Data Compatibility / Standardization #2

[cardoe]

Let me expand on this since I suggested it on the Reddit thread. It plays into #2 that you posted. I'm a big proponent of the UNIX model. Separate utilities / interfaces so that others can glue them together in new and interesting ways. By creating clean separations, you'll be able to develop quicker and bring in new contributors more easily. People will be interested in different aspects and will want to scratch their itch. If they have to build a large project or attempt to digest a large code base it will be a barrier to bringing new contributors into the project. Lastly you'll be able to utilize the beancount ledger file test suite to ensure correctness of the parsing crate. At the end of the day, you'll need to be able to parse the data files before you can do any other CLI operations.

[alensiljak]

It plays into #2

Couldn't agree more. A Rust crate can be utilized from C/++, Python, JavaScript, and what-not. For standards-compatible application, this would heaven-sent.
I wonder how useful this would be in case someone wants to expand on the syntax.

I agree with most your points. but would not consider this to be "my" project. I see myself mostly as a steward, pooling the resources together and doing what I can. I certainly also want to help with coding even though I'm a Rust beginner.
In terms of direction, I'd follow advice of more experienced people, like John himself.

[tbm]

While I'm not terribly interested in a Rust re-write of ledger, I am very interested in a standalone parser for ledger written in Rust.

I briefly looked around and found nom which looks very interesting. I am not a good programmer and I don't know Rust, but I'd be happy to test such a parser.

FWIW, I believe ledger2beancount has the most comprehensive test suite for ledger syntax and I'd be happy to contribute this test suite to a Rust parsing library.

There should be a separate parser for hledger because the syntax is slightly incompatible with ledger.

[alensiljak]

Thanks! The parser as an independent component is certainly a good idea and can be used at various places. I would like to use it in Cashier (a progressive web application) when compiled to WASM, for example.
Adding parsers for different (dialects of) Plain-Text Accounting formats may be a good idea! As Beancount has already moved on the path of using a compiled language for implementation of performance-critical components, Rust may be a good option down the road, too. Plus these could be reusable among different projects (hopefully!).

[dantuck]

There is already a pretty good ledger parser that could be contributed too and pulled into this project.

[alensiljak]

There is already a pretty good ledger parser that could be contributed too and pulled into this project.

Now that I'm digging a bit more into parsing, I see that it is based on nom. I'll open a separate discussion on parsers and keep this one for the separate crate / library only.

[antage]

I wrote ledger-dialect (not compatible) parser with chumsky crate: grossbuch.
Maybe this will be helpful.

[alensiljak]

Thank you! Let's see where the path leads in the near future and collaborate on related tools in the PTA and Rust ecosystem(s).

[alensiljak]

I have created a page with the list of resources that may be useful for the parser, collecting the info from this discussion:
https://github.com/ledger-rs/ledger-cli-rs/wiki/Parser-Resources

[alensiljak]

What makes your parser incompatible? Is it incomplete or did you have a different purpose for it?

[antage]

It's my experimental project. You can see examples of the syntax.

[alensiljak]

Should an existing Ledger parser be extracted first or should a new one be written using an existing parser crate/library (like nom or chumsky)?

[antage]

Porting the ledger parser line by line can be difficult due to a borrow checker. I see that Ledger uses LL(*) parser, so chumsky is the natural choice for a reimplementation.

[zmrow]

Before anyone gets too down and dirty or in the weeds with a low level parser like nom, what about using serde and writing a deserializer? The framework and method is well established and the pattern is pretty standard in the community

[antage]

Serde does not automagically deserialize. You need a parser anyway.

[35VLG84]

Hello,

FYI, Tackler has ANTRL4 based parser for simplified Ledger syntax . The parser definition is located here.

• parser: https://gitlab.com/e257/accounting/tackler/-/blob/main/core/src/main/antlr4/TxnParser.g4
• lexer: https://gitlab.com/e257/accounting/tackler/-/blob/main/core/src/main/antlr4/TxnLexer.g4

This is for java/scala, but it should be possible to generate parser definition for Rust target also. There is an experimental Rust target for ANTRL4 here: https://github.com/rrevenantt/antlr4rust

I have been very happy working with ANTRL4 and the produced parser is fast: https://tackler.e257.fi/docs/performance/