[ZK Stack] - Is witness-generator reading from the wrong storage location?

[moneta]

Team or Project

No response

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

1.85.0

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

I deployed the proof-data-hander and all other sequencer components to run on separate servers. These servers share the same the volume folder to store the artifacts folder of the sequencer node.

Then I have the prover gateway running on another server that talks with the proof-data-handler. The prover gateway saves its witness input files in the GCS. All works perfectly.

However, when I deployed the witness-generator, it tries to read the witness input data bin file in the cloud bucket, while these files are stored in the artifact folder.

I am using code from the branch core-v27.3.0. Is it the bug?

Here are my storage config in the general.yaml:

core_object_store:
      file_backed:
      file_backed_base_path: ./chains/revolution/artifacts/
prover_object_store:
      gcs_with_credential_file:
      mode: GCSWithCredentialFile
      bucket_base_url: dev-revo-public
      gcs_credential_file_path: /root/.config/gcloud/application_default_credentials.json

Here is the error logs:

2025-06-03T20:33:31.339826Z  INFO zksync_witness_generator::rounds: Processing BasicCircuits job 1                                                                                                            
2025-06-03T20:33:31.339876Z  INFO zksync_witness_generator::rounds::basic_circuits: Processing FRI basic witness-gen for block 1                                                                              
2025-06-03T20:33:31.751898Z ERROR zksync_task_management: One of the tokio actors unexpectedly finished with error: get_next_job(): prepare_job(): Getting data with bincode failed with key not found: HTTP s
tatus client error (404 Not Found) for url (https://storage.googleapis.com/storage/v1/b/dev-revo-public/o/witness_inputs%2Fwitness_input_data_1.bin?alt=media), getting data with CBOR failed with fatal error
 accessing object store: HTTP status client error (403 Forbidden) for url (https://storage.googleapis.com/storage/v1/b/dev-revo-public/o/witness_inputs%2Fwitness_inputs_1.cbor?alt=media)                    
2025-06-03T20:33:31.752140Z  WARN zksync_queued_job_processor: Stop signal received, shutting down fri_node_aggregation_witness_generator component while waiting for a new job                               
2025-06-03T20:33:31.752169Z  WARN zksync_queued_job_processor: Stop signal received, shutting down fri_scheduler_witness_generator component while waiting for a new job                                      
2025-06-03T20:33:31.752190Z  WARN zksync_queued_job_processor: Stop signal received, shutting down recursion_tip_witness_generator component while waiting for a new job                                      
2025-06-03T20:33:31.752196Z  WARN zksync_queued_job_processor: Stop signal received, shutting down fri_leaf_aggregation_witness_generator component while waiting for a new job                               
2025-06-03T20:33:31.752671Z  INFO zksync_witness_generator: Finished witness generation    

Note:

• Files in the GSC storage bucket: witness_inputs_1.cbor
• Files in the artifact folder on the shared volume: vm_run_data_1.cbor

Expected Behavior

The witness generator component should find the file vm_run_data_1.cbor from the artifact folder where the file exists.

# ls -la chains/revolution/artifacts/witness_inputs/
total 978
drwxr-xr-x  2 root root      2 Jun  2 22:50 .
drwxr-xr-x 14 root root     12 Jun  2 22:50 ..
-rw-r--r--  1 root root 105368 Jun  2 22:50 merkle_tree_paths_1.cbor
-rw-r--r--  1 root root 895685 Jun  2 22:50 vm_run_data_1.cbor

Current Behavior

The witness generator component is finding the file vm_run_data_1.cbor in the google storage bucket

Repository Link (if applicable)

No response

Additional Details

No response

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.

[EmilLuta]

Hey @moneta, thanks for reaching out. Most likely, the issue is related to witness generator looking for the .bin file (due to older version). Prover & Core have different versioning and bumping prover further might solve your problem (albeit, can't tell what to from your logs).

Looking through your setup, what I find confusing is that you use BOTH GCS & local artifacts folder. Did you mean to use one or the other?

[moneta]

Yes, I do use both local artifacts folder and GCS. The local folder is for the sequencer components and GCS for prover components.

[EmilLuta]

I see. And what do you have under witness_inputs folder in GCS?

[moneta]

I have only the witness_inputs_1.cbor

[moneta]

I uploaded files in the artifacts folder to GCS and it works. So you can close this issue or send it as a bug to the prover team in case of mix storage configuration.

[EmilLuta]

Sorry for the late reply.

I wonder if your prover gateway config is file based, instead of GCS. Data is transferred from sequencer (known as core internally) to prover via proof_data_handler (from core) to prover gateway (from prover). As far as I can tell, the prover gateway uses the correct configuration (can be seen here).

Nonetheless, happy to hear the problem is solved!