[ZK Stack] - prover-job-monitor does not recover from compressor failure

[thecodingshrimp]

Team or Project

No response

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

1.87.0

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

When processing a proof and the compressor fails to compress the proof (in my case, the gpu was 100% occupied by the prover), the prover-job-monitor does not reset/update the state of the current compressor job. When restarting the compressor/prover-job-monitor the job is not picked up again.

Expected Behavior

compressor/job-monitor update the state of the compressor job as failed and, in a best case scenario, reschedule it for a later time. when restarting the compressor, it should handle the failed job/in progress job.

Current Behavior

job-monitor still thinks the job is in progress, compressor does not pick up job after restarting nor updates the state of the job.

Repository Link (if applicable)

No response

Additional Details

No response

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.

[JackHamer09]

@thecodingshrimp hey, thanks for reporting this problem. Would you be able to provide details on how to reproduce this issue?

[JackHamer09]

I see that you provided some in another thread. These are still different issues right?

[thecodingshrimp]

they are different issues, yes.
I think I can close it for now since I noticed that after 5 attempts the compressor gives up on the batch. The monitor probably will stop reporting it by then too. let me double check and reopen the issue if necessary.