[ZK Stack] - "invalid sender" error from eth client when sending txs

[SergeevDmitry]

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

1.74.0-nightly

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

I have ran local hyperchain based on docker setup and using prover setup (CPU) of core-v23.10 version.

To make it all work, I had to make some changes:

1. I noticed that the witness-vector-generator sends requests to http://metadata.google.internal/computeMetadata/v1/instance/zone which is not accessible from other hosting providers. Is it intentional and ZK Stack should be run only on google cloud? This produces a panic and the service cannot start. I hardcoded zone to general value and after that the service started without any errors. Is it normal and does any code depend on this value?
2. After that zkstack-witness-generator-basic-circuits service started to produce errors that it cannot get L1BatchWithMetadata, because commitment and other required batch fields are empty in the database. After digging around I found that commitment_generator component should be enabled which is not by default in the hyperchain docker compose file.

My current problem is that zkstack-core service tries to send transactions to the ValidatorTimelock contract on L1, but gets "invalid sender" error from eth client. All transactions are signed by ETH_SENDER_SENDER_OPERATOR_BLOBS_ETH_ADDR.

Expected Behavior

Transactions are sent to the L1 successfully with correct signature.

Current Behavior

Transactions get rejected by eth client

Repository Link (if applicable)

No response

Additional Details

zkstack-core  | 2024-05-08T15:46:27.897019Z  WARN loop_iteration{previous_block=L1BlockNumber(352373)}: zksync_core::eth_sender::eth_tx_manager: Error when sending new signed tx for tx 1, base_fee_per_gas 10, priority_fee_per_gas: 1000000000: Ethereum gateway Error Request to ethereum gateway failed: RPC error: Error { code: ServerError(-32000), message: "invalid sender", data: None }
zkstack-core  | 2024-05-08T15:46:27.926933Z  WARN loop_iteration{previous_block=L1BlockNumber(352373)}: zksync_core::eth_sender::eth_tx_manager: Error when sending new signed tx for tx 2, base_fee_per_gas 10, priority_fee_per_gas: 1000000000: Ethereum gateway Error Request to ethereum gateway failed: RPC error: Error { code: ServerError(-32000), message: "invalid sender", data: None }

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.

[bxpana]

Thanks for posting. Will have the team take a look and get back to you.

[SergeevDmitry]

Hello. Any updates?

[bxpana]

Apologies for the delay here. Let me check on that and see if I have any updates

[bxpana]

No update on this yet, but we're working on getting to this.

[SergeevDmitry]

Hello. The problem was with wrong chain_id that is used in transactions. I use personal L1 network, but as I found the list of available L1 networks is hardcoded and it is not possible to set it via config - https://github.com/matter-labs/zksync-era/blob/main/core/lib/basic_types/src/network.rs#L18. I had to add my network to the list. I'm sorry, I did not mention it, because I thought that the chain_id is configured by ETH_CLIENT_CHAIN_ID variable.

Now I see 2 errors in the zkstack-witness-generator-basic-circuits service.

zkstack-witness-generator-basic-circuits-1  | Running out of circuit for 4294967295 cycles
zkstack-witness-generator-basic-circuits-1  | Made INITIAL at cycle 1024
zkstack-witness-generator-basic-circuits-1  | thread 'tokio-runtime-worker' panicked at /usr/local/cargo/git/checkouts/era-zk_evm-e18c28df0674bd3f/6250dbf/src/opcodes/execution/log.rs:110:21:
zkstack-witness-generator-basic-circuits-1  | refund can not be more than net cost itsel

zkstack-witness-generator-basic-circuits-1  | thread 'tokio-runtime-worker' panicked at witness_generator/src/basic_circuits.rs:759:64:
zkstack-witness-generator-basic-circuits-1  | called `Result::unwrap()` on an `Err` value: JoinError::Panic(Id(370), ...)

I'm not sure if it is related, but when I ran zk stack prover-setup command it generated setup_2^24.key file and I renamed it to setup_2^26.key, because it is copied into docker container here - https://github.com/matter-labs/zksync-era/blob/main/docker/proof-fri-compressor/Dockerfile#L20

[Artemka374]

The issue seem to be related to mismatching versions of VM in core/prover. We have different releases of prover and core components, so that's where you caught this mismatch. In revision core-v23.1.0 which you are using, core VM is v1.5.0, while provers are using VM v1.4.1. As a solution I recommend switching to tag prover-v14.0.0 where these versions are matching

[SergeevDmitry]

Hello. Thank you for the answer. The tag you've suggested has broken CLI tool and I can't configure my network. Probably I should wait for a new release at the beginning of June. I saw somewhere that CLI tool will start to work in this release