Skip to content

OOM when runing CPU prover #2223

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
2 tasks done
nuke-web3 opened this issue Apr 8, 2025 · 4 comments
Open
2 tasks done

OOM when runing CPU prover #2223

nuke-web3 opened this issue Apr 8, 2025 · 4 comments
Assignees
Labels
bug Something isn't working SP1

Comments

@nuke-web3
Copy link

nuke-web3 commented Apr 8, 2025

Component

Other (please describe)

Have you ensured that all of these are up to date?

  • SP1 SDK
  • cargo prove CLI/sp1up

What version of SP1 SDK are you on?

sp1-zkvm = "4.0.0"

What version of the cargo prove CLI are you on?

cargo-prove sp1 (a80c17c 2025-04-01T00:41:38.259051986Z)
cargo 1.86.0 (adf9b6ad1 2025-02-28)

Operating System

None

Describe the bug

I have been building a demo around SP1 and testing full proving to bench on two separate machines that exhibit the same behavior: RAM is saturated and swap is then saturated.

For details, see nuke-web3/sp1-chacha#1

Note: I kill the process manually before swap actually fill up to avoid true OOM issues

Example behavior monitored by btm

Image

See above for basic system specs, notably 32 GB RAM + 20 GB swap.

Reproduce

With SP1 installed:

# open some system monitor like btm or top
git clone https://github.com/nuke-web3/sp1-chacha.git
cd sp1-chacha
cargo r -r -- --prove
# BE SURE TO WATCH RAM & SWAP USE!
@leruaa
Copy link
Contributor

leruaa commented Apr 15, 2025

@nuke-web3 Do you still have this issue?

@nuke-web3
Copy link
Author

Yes I do, sadly. No changes on my side for behavior. Happy to help test more things if that is helpful.

What machine(s) did you try to reproduce on? (CPU, RAM, Swap, etc.)

@leruaa
Copy link
Contributor

leruaa commented Apr 15, 2025

Oh you are right, I see ~100GB of memory usage. It don't OOM because I have 500GB on the server it's running.

@nuke-web3
Copy link
Author

So it eventually does approach some maximum? I was not able to see it levelize.

I was told that SP1 should be aware of the env it is in and not fully use RAM... maybe that isn't the case?

@ratankaliani ratankaliani self-assigned this Apr 29, 2025
@linear linear bot added bug Something isn't working SP1 labels May 11, 2025
@linear linear bot assigned leruaa and unassigned ratankaliani May 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working SP1
Projects
None yet
Development

No branches or pull requests

3 participants