[ZK Stack] - Zkstack ecosystem deployment stability issue

[mikiw]

Team or Project

GlitchD Labs

ZK chain

My own chain

Environment

Testnet

L2 block number

No response

zkSolc Version

I use this locked version cargo install --git https://github.com/matter-labs/zksync-era/ --locked zkstack --force --rev 3e9fc0a5f6abffdae994e4f22ab21eac95c7ac08

zksync-ethers Version

I use this locked version cargo install --git https://github.com/matter-labs/zksync-era/ --locked zkstack --force --rev 3e9fc0a5f6abffdae994e4f22ab21eac95c7ac08

Minimal Reproducible Code or Repo Link

It's hard to reproduce the problem. Sometimes deployment is working fine, and sometimes it is not.

Does this work on other EVMs? (If yes, please list at least 1 of them)

not applicable

Description of What Your Contract Does

It's a script to automate zkstack ecosystem deployment.

Additional Details

During the deployment ecosystem init phase, zkstack ecosystem init, the deployment is sometimes working and sometimes failing both on holesky and sepolia. At the beginning, I was thinking that this was related to the RPC node, but it is not. I already tested this with many nodes (public, infura, alchemy, both paid and free) on both Holesky and Sepolia, and the results are the same.

For successful deployment, we need 68 deployer transactions, 25 governor transactions, and 6 chain governor transactions. Sometimes it's just a couple of deployer transactions (sometimes 11, sometimes 50, etc.) and no governor or chain governor transactions.

◒  Accepting governance...                                                      Command: forge script deploy-scripts/AcceptAdmin.s.sol --legacy --ffi --rpc-url=https://ethereum-holesky-rpc.publicnode.com --broadcast --sig=7475e9ea0000000000000000000000008aaa083a9c744ae4146124576b97cf8e5ba9b5c1000000000000000000000000b5b24d4690ecd5c4e61e9d812e8f133db6d6bebc --private-key=da603224bacec746728d4366cfdfbac43794ff8bb18be3b96ecbff6953dd57ac
◒  Accepting governance...                                                      │
■  Command failed to run
│  
■    Status:
│      exit status: 1
│    Stdout:
│      No files changed, compilation skipped
│      Script ran successfully.
│      
│      ## Setting up 1 EVM.
│      
│      ==========================
│      
│      Chain 17000
│      
│      Estimated gas price: 0.001023216 gwei
│      
│      Estimated total gas used for script: 96445
│      
│      Estimated amount required: 0.00000009868406712 ETH
│      
│      ==========================
│      
│      Transactions saved to:
│      /home/ubuntu/zk_stack/zksync-era/contracts/l1-contracts/broadcast/AcceptAdmin.s.sol/17000/7475e9ea-latest.json
│      
│      Sensitive values saved to:
│      /home/ubuntu/zk_stack/zksync-era/contracts/l1-contracts/cache-forge/AcceptAdmin.s.sol/17000/7475e9ea-latest.json
│      
│      
│  
│    Stderr:
│      Error: Failed to send transaction
│      
│      Context:
│      - server returned an error response: error code -32000: nonce too low: next nonce 7, tx nonce 5
│      
│  
■  Command failed to run: forge script deploy-scripts/AcceptAdmin.s.sol --legacy --ffi --rpc-url=https://ethereum-holesky-rpc.publicnode.com --broadcast --sig=7475e9ea0000000000000000000000008aaa083a9c744ae4146124576b97cf8e5ba9b5c1000000000000000000000000b5b24d4690ecd5c4e61e9d812e8f133db6d6bebc --private-key=da603224bacec746728d4366cfdfbac43794ff8bb18be3b96ecbff6953dd57ac
│  Error: Failed to send transaction
│  
│  Context:
│  - server returned an error response: error code -32000: nonce too low: next nonce 7, tx nonce 5
│  
│  
▲    0: Command failed to run: forge script deploy-scripts/AcceptAdmin.s.sol --legacy --ffi --rpc-url=https://ethereum-holesky-rpc.publicnode.com --broadcast --sig=7475e9ea0000000000000000000000008aaa083a9c744ae4146124576b97cf8e5ba9b5c1000000000000000000000000b5b24d4690ecd5c4e61e9d812e8f133db6d6bebc --private-key=da603224bacec746728d4366cfdfbac43794ff8bb18be3b96ecbff6953dd57ac
│  
└  Failed to run command
send: spawn id exp3 not open
    while executing
"send "postgres://postgres:notsecurepassword@localhost:5432\r""

[mikiw]

│ Transactions saved to:
│ /home/ubuntu/zk_stack/zksync-era/contracts/l1-contracts/broadcast/AcceptAdmin.s.sol/17000/7475e9ea-latest.json

{
  "transactions": [
    {
      "hash": null,
      "transactionType": "CALL",
      "contractName": null,
      "contractAddress": "0x8aaa083a9c744ae4146124576b97cf8e5ba9b5c1",
      "function": "multicall((address,uint256,bytes)[],bool)",
      "arguments": [
        "[(0xB5b24D4690EcD5c4e61e9d812e8F133dB6D6BeBc, 0, 0x0e18b681)]",
        "true"
      ],
      "transaction": {
        "from": "0x924bf64a7523ab37ee589159fb211091b4760a8a",
        "to": "0x8aaa083a9c744ae4146124576b97cf8e5ba9b5c1",
        "gas": "0x178bd",
        "value": "0x0",
        "input": "0x69340beb0000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000020000000000000000000000000b5b24d4690ecd5c4e61e9d812e8f133db6d6bebc0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000040e18b68100000000000000000000000000000000000000000000000000000000",
        "nonce": "0x5",
        "chainId": "0x4268"
      },
      "additionalContracts": [],
      "isFixedGasLimit": false
    }
  ],
  "receipts": [],
  "libraries": [],
  "pending": [],
  "returns": {},
  "timestamp": 1745494997,
  "chain": 17000,
  "commit": "fc26d9fc0"
}

Successful transactions on Holesky:

Deployer address: 0x2a579e8387d55d4e362e43119849c406f64c8c74 [25 of 68]
https://holesky.etherscan.io/address/0x2a579e8387d55d4e362e43119849c406f64c8c74

Governor address: 0x924bf64a7523ab37ee589159fb211091b4760a8a [8 of 25 transactions]
https://holesky.etherscan.io/address/0x924bf64a7523ab37ee589159fb211091b4760a8a

Chain Governor address: 0x25328aa7ea195fcad1985619f5b1a059bd61f2e2 [0 of 6 transactions]
https://holesky.etherscan.io/address/0x25328aa7ea195fcad1985619f5b1a059bd61f2e2

[ly0va]

Hello @mikiw! The variable number of deployer transactions is explained by the fact that the deployment is performed via CREATE2, which makes it easy to check if a certain contract has already been deployed by the deployer, and if so, does not redeploy it, and thus does not create separate transaction on chain.

The "nonce too low" error may stem from the fact that you might have re-initiated the zkstack ecosystem init process before waiting for all the transactions from the previous invocation to be settled on chain. This would have resulted in a race condition, where:

• some transactions from the old invocation still in mempool but not executed
• which makes the provider return a low nonce in the new invocation
• then, before the transactions from the new invocation are accepted, the transactions from the old invocation are executed, increasing the nonce
• which makes the transactions from the new invocation get rejected because of low nonce

To fix, do either of:

• wait until all of the transactions from the old invocation of zkstack ecosystem init are executed
• change the deployer wallet
• set the nonce manually

Hope this helps, feel free to unmark this discussion as answered if you have further questions

[mikiw]

Hi @ly0va,
Thank you for your response! However, it seems that the script is executed only once.

Sometimes the errors are different, for example:

◒  Accepting governance...                                                      Command: forge script deploy-scripts/AcceptAdmin.s.sol --legacy --ffi --rpc-url=https://eth-holesky.g.alchemy.com/v2/zdU955TmIXH9xSSSfLMj9yC4GPQKwfH0 --broadcast --sig=83eddd190000000000000000000000009cd2e1a43d8e4810d29a078fe78cf84ebf995c2e000000000000000000000000d1603c6cd50a6ed7444b406bb445db9d9b332c34 --private-key=ca2bd15852532898a8dbd7bc0a0fda82fa05df8353f55cdddee01720e9a4eb81
◐  Accepting governance...                                                      Proceed with the deployment anyway
◓  Accepting governance...                                                      │
■  Command failed to run
│  
■    Status:
│      exit status: 1
│    Stdout:
│      
│  
│    Stderr:
│      Error: contract source info format must be `<path>:<contractname>` or `<contractname>`
│      
│  
■  Command failed to run: forge script deploy-scripts/AcceptAdmin.s.sol --legacy --ffi --rpc-url=https://eth-holesky.g.alchemy.com/v2/zdU955TmIXH9xSSSfLMj9yC4GPQKwfH0 --broadcast --sig=83eddd190000000000000000000000009cd2e1a43d8e4810d29a078fe78cf84ebf995c2e000000000000000000000000d1603c6cd50a6ed7444b406bb445db9d9b332c34 --private-key=ca2bd15852532898a8dbd7bc0a0fda82fa05df8353f55cdddee01720e9a4eb81
│  Error: contract source info format must be `<path>:<contractname>` or `<contractname>`
│  
│  
▲    0: Command failed to run: forge script deploy-scripts/AcceptAdmin.s.sol --legacy --ffi --rpc-url=https://eth-holesky.g.alchemy.com/v2/zdU955TmIXH9xSSSfLMj9yC4GPQKwfH0 --broadcast --sig=83eddd190000000000000000000000009cd2e1a43d8e4810d29a078fe78cf84ebf995c2e000000000000000000000000d1603c6cd50a6ed7444b406bb445db9d9b332c34 --private-key=ca2bd15852532898a8dbd7bc0a0fda82fa05df8353f55cdddee01720e9a4eb81
│  
└  Failed to run command
send: spawn id exp3 not open
    while executing
"send "Proceed with the deployment anyway\r""