[Dev Tools] anvil-zksync replay_tx not matching chain data

[coffeexcoin]

Team or Project

Abstract

ZK chain

Abstract

Environment

Mainnet

Select the Dev Tool you are using

Foundry

Provide the version of the tool (if applicable)

0.6.3

Provide a brief description of the functionality you're trying to implement and the issue you are running into.

I am attempting to debug and understand the [Panic] error in the failing delegatecall frame from transaction 0xebd778429b226583a65b8bfb165ca7b3ab6218e3c73e2bc9e5ed6bb36dbeb647 by comparing against a similar, succeeding transaction 0x99a0c5d543f2b44d8e80349dec54f05a9ad37b07b9dec7a92ae4a0b4d73ed674

To attempt to understand the differences in call flow I run the following commands:

Failing TX

anvil-zksync -vv replay_tx --fork-url abstract 0xebd778429b226583a65b8bfb165ca7b3ab6218e3c73e2bc9e5ed6bb36dbeb647

Successful TX

anvil-zksync -vv replay_tx --fork-url abstract 0x99a0c5d543f2b44d8e80349dec54f05a9ad37b07b9dec7a92ae4a0b4d73ed674

Outputs here:
https://gist.github.com/coffeexcoin/bac1a75b14150ec3d548c59c56b57531

As you can see, both replays result in the following error frame, despite there not being any error frames in the successful transaction.

│  └─ ← [Halt: TracerCustom("Panic")]
└─ ← [Revert: Unknown revert reason]

Additionally, when run with an additional verbose flag (-vvv) both transactions are shown emitting an "UnreceivedTokenCached" event, even though the succeeding transaction never emitted such an event onchain (the successful one emitted the expected OFTReceived event.

Anvil success TX

├─ emit UnreceivedTokenCached(: 0xcaa1a0f3e38a4abf5fbe086adb8a1963dd483446d36cb8a470f03d4ddf7f5154, : 0, : 30340, : [0x519d06fa038f36f88c616c2aae1006ad9abaf0a1], : 399000000000000, : 0x)

Anvil failed TX

├─ emit UnreceivedTokenCached(: 0x1c06230299cbceaca45d905fa749cbfa0092c93b22d9bfd058fe342c4c524968, : 0, : 30101, : [0x8e69c3451fe32ebf64f4c23748ecb818254253c0], : 260011000000000000, : 0x)

Any insight on how to troubleshoot this partial failure/panic would be greatly appreciated (as well as proper usage of zksync-anvil for doing so)

Repo Link (Optional)

No response

Additional Details

No response

[cpb8010]

This is a tough one, adding a follow up here: matter-labs/anvil-zksync#727

[itegulov]

I did a small investigation and the reason why execution in anvil-zksync is different is that, unfortunately, gas consumption when replaying an individual transaction (or even an individual block) is different from what it is in real life execution. For instance, it's a lot cheaper to write into slots that have already been written into by some previous transaction in the same batch.

To fully replicate what happened in real life, anvil-zksync would have to re-execute all transactions from that batch. But batches live for quite a while on the majority of ZK chains (including Abstract) and it is unreasonable for anvil-zksync to replay all of them. For example, second transaction from the original post was included in batch 20497 which has 2987 transactions. If I had to guess, it would have taken a couple of hours to execute all of them (assuming RPC would not have rate limited you halfway through). So, unfortunately, there is no reasonable solution short-term but things will get better with Boojum OS though as batches AFAIU will become short-lived there.

For transaction in question: transfer limit set in 0x221f0e1280ec657503ca55c708105f1e1529527d is 15000 which, as you can see, is too conservative for some cases. Setting the limit to 30000 and then replaying both transactions makes both of them successful in anvil-zksync. Final gas consumption for the transfer in both cases is just over 25000.