(black-box) Testing

[alensiljak]

How to ensure full compatibility of the rewrite with the original Ledger?
A black-box testing tool is needed, to run the same set of tests on both executables. ✅
While creating a test harness, run on existing ledger with the test data files.

Goals

The most important group of tests would be Acceptance Tests.
Their purpose would be to:

1. test all of the Ledger's functionality
2. ensure that the new Rust application matches this

Test Suite

The Acceptance test suite should use the sample set of journal files, run it through ledger and rledger (the new app) and assert that their output matches.

Cargo Test can be used as the test runner. Additional crates may help with this task.
A CLI testing suite can run any CLI application. => https://github.com/alensiljak/cli-test-executor was written for this purpose. It supports existing Ledger's test syntax.

Need to identify how to compare the output - identical output from both apps, or comparing the features through an output parser, API output comparison, etc.

Other

Additional resources/notes:
https://github.com/ledger-rs/incubator/wiki/Testing-Resources

To-Do

• check existing test suites
  • ledger2beancount => comprehensive syntax samples. Use the .ledger samples. The parser should process all these cases.
  • ledger
    • python tests
    • ctest
• complete the CLI test runner
• go through the Ledger documentation to identify any missing test cases
  • add any missing tests
• create a repository for the Acceptance Tests => https://github.com/ledger-rs/acceptance-tests
  • collect existing tests here. Standalone binaries can be built to run independently.
• define test data files
  • use ledger2beancount *.ledger files as input. Keep in a separate directory for easy comparison with the original location.
  • individual cases with a small number of transactions that focus on a specific aspect. Any future bug should be documented in tests as well.
  • a big file with everything? => No, use focused test data.
• define a test suite
  • adapt ledger test files
  • create tests for l2b data files
  • set-up stress tests (see https://github.com/orgs/ledger-rs/discussions/29#discussioncomment-4721794)
• find a test runner that can run a cli => Rust integration tests + additional crates
• define how to compare outputs (as the output of the new version might not match the original)
  • output parser? (might also be useful for other tools that read the output, like Cashier)
• generate standalone test binaries (i.e. Ledger regression)

[alensiljak]

Perhaps, using the Rust toolchain, a set of Integration tests can be written. These are calling an external process, which can be manually modified (in a config file?) to use a specific executable - ledger-cli or ledger-cli-rs.

[alensiljak]

https://rust-cli.github.io/book/tutorial/testing.html

[alensiljak]

Martin recommends checking ledger2beancount for its comprehensive test suite.

Findings:
Added to the list. The input files are extremely valuable. They contain the majority of Ledger-supported syntax that the parser should process.

[alensiljak]

Ledger has a set of tests in https://github.com/ledger/ledger/tree/master/test

• Analyze if it can be reused
• select acceptance tests

Findings so far:
Some Ledger performance tests are utilizing Valgrind and seem to run on Linux only.
The business logic tests seem to be run with Python.
acprep is a Python script that runs the tests, among other things. Doesn't run out-of-the-box on Windows or WSL (with a clone on NTFS partition). Requires a full build.

[cardoe]

I don't think you want a line for line exact output match to ledger, hledger, or beancount. You'll end up having / wanting your own interface. This will allow you to truly provide a reason to use this tool. What I believe you want is file format compatibility which I mentioned in #11.

[alensiljak]

That's a good point. I was mostly jotting down the ideas I had, not necessarily working them out in detail. This is what other pairs of eyes are for, obviously. ;)
Yes, that's correct. Obviously the same text output is not the goal, but the same semantic output. I.e. given the input file, the balances on these accounts has to be x, y, z. That type of thing.

[dantuck]

I would imagine we would have many differences in this new tool that will help it standout from ledger-cli. I agree with @cardoe and we should more so check compatibility of feature parity so one can take their existing ledger file and use it between the two cli's without getting different calculations in the output.

[alensiljak]

I am currently trying-out Ledger's test harness. I think, between that and ledger2beancount input data collection, we would have a decent coverage (in input files at least!) of the required functionality. The first priority of the application would probably be to have (major) compatibility with existing data files. I'm imagining starting a separate repository soon, just for the Acceptance Tests. So no matter what the end output of the application development is, the tests should still work with any CLI. I'd like the test system to be a bit generic. Some Ledger tests are nice in that they contain the data, the command, and the output. Structured like that, there's no need to have much logic in the test code itself.
Any opinion or an example of a similar process are more than welcome, btw!

[alensiljak]

To check: https://en.wikipedia.org/wiki/Expect#Rust

[alensiljak]

This is useful for interaction with CLI apps. Not required here, in general, since the ledger commands are a one-off.

The REPL mode could be interacted with using rexpect library/crate, for example.

[simonmichael]

hledger tests use shelltestrunner, a tool inspired by Ledger's functional tests with a little more flexibility - might be of interest. Coincidentally I recently converted Ledgers func tests to shelltests. Thanks for mentioning ledger2beancount, I'll look at those also.

[alensiljak]

Thanks, Simon! Let's see if we can minimize duplication and have something useful for a broader community. I'm still looking/testing what's out there. Outlined here - https://github.com/orgs/ledger-rs/discussions/6#discussioncomment-4470846
I'll add shelltestrunner to my list of things to check out. Do you have a link to the tests you converted?
I was thinking of creating a separate repo that would focus only on tests. I.e. some simple template format - data, command, output. That would work for any CLI application.

Edit: alright, that's exactly what shelltestrunner does!

[alensiljak]

The ledger2beancount has a great collection of input files. These are not tests as-such but are good for testing parsers directly and they can be used as a base for defining an output. Meaning, the commands and the output need to be defined to test the functionality of the app.
But it's a great comprehensive set of journal syntax. It may be useful to collect these independent of a specific project, as they are really useful and should be contributed to if/when a bug is encountered in the future.

[simonmichael]

https://github.com/simonmichael/hledger/tree/master/hledger/test/ledger-compat <https://github.com/simonmichael/hledger/tree/master/hledger/test/ledger-compat> > baseline, regress are a snapshot of Ledger's main func. tests with the actual tests commented out and replaced by a hledger read test. And for completeness, https://github.com/beancount/ledger2beancount/tree/master/tests <https://github.com/beancount/ledger2beancount/tree/master/tests> is ledger2beancount's test files.

[alensiljak]

Got it! Added to https://github.com/ledger-rs/incubator/wiki/Testing-Resources, together with a sample syntax for multiple tests.

I'm still not certain what the best approach would be in feature comparison. This is something that may also be interesting for other applications (i.e. hledger). It would be simple if the output was identical. But, if not (which is most likely), some kind of automatic result comparison might need to be established. Either a parser or just manual rewrite of the output with the given input and command. This duplicates the tests, though, and there's no direct relation between ledger and x tests, except by name.

[alensiljak]

Updated the description and the tasks overview in the header post.