Investigate the performance regression in onchain decider

[winderica]

The onchain decider SNARK without the light-test feature now takes longer time and more RAM. In fact, the setup and proof generation could be done with 32GB of RAM in 2 minutes, as reported in #88. However, the onchain decider now takes more than 3 minutes and eventually fails to run on my machine with 64GB of RAM:

> cargo test --package folding-schemes --lib --release -- folding::nova::decider_eth::tests::test_decider --exact --nocapture
    Finished `release` profile [optimized] target(s) in 0.19s
     Running unittests src/lib.rs (target/release/deps/folding_schemes-850178000889a0b5)

running 1 test
Start:   KZG10::Setup with degree 65536
··Start:   Generating powers of G
····End:     Generating powers of G ................................................87.030ms
··Start:   Generating powers of gamma * G
····End:     Generating powers of gamma * G ........................................91.847ms
··Start:   Generating negative powers of h in G2
····End:     Generating negative powers of h in G2 .................................296ns
··End:     KZG10::Setup with degree 65536 ..........................................182.324ms
Start:   Groth16::Generator
··Start:   Constraint synthesis
····End:     Constraint synthesis ..................................................17.176s
··Start:   Inlining LCs
test folding::nova::decider_eth::tests::test_decider has been running for over 60 seconds
error: test failed, to rerun pass `-p folding-schemes --lib`

Caused by:
  process didn't exit successfully: `/home/winderica/ivc/folding-schemes/target/release/deps/folding_schemes-850178000889a0b5 'folding::nova::decider_eth::tests::test_decider' --exact --nocapture` (signal: 9, SIGKILL: kill)

[pmikolajczyk41]

hi @winderica , do you have any updates on this one?

[winderica]

Thanks for bringing this up @pmikolajczyk41!

I once looked into this and found that it might be caused by #203. This is pretty weird because the PR decreased the number of constraints, so the memory cost should be lower.

My only suspicion was that the joint_scalar_mul_be might introduce deeply nested linear combinations that blew up the stack, but I have been busy with some other stuff and forgot to investigate further...

Will try to take a deeper look into it these days, but I can't promise anything :)

[pmikolajczyk41]

I have just run one of the examples (external_inputs), slightly modified, on AWS c7a.16xlarge instance. Parameter generation takes almost 90GB of RAM and a few minutes to finish. Anyway, I got SNARKVerificationFail - however, with light-test feature on, everything works smoothly. Might be that I messed something, but anyway I guess the problem needs investigation.

If you need any help or somebody to take it over from some point, I'd be happy to do so. Just let me know.

[winderica]

Anyway, I got SNARKVerificationFail - however, with light-test feature on, everything works smoothly.

Thanks for providing the result of a complete run! It is strange that not only the memory cost became much larger, but the verification also eventually failed, because the optimized circuits in #203 passed the unit test.

I just created a branch that reverts #203, and the decider verification was successful on my machine within 30 GB of RAM without the light-test feature. So please use this branch as a temporary workaround until I find the root cause!

[winderica]

Sorry for the long delay. I just finished #227, which should be sufficient for fixing this issue!