Issue tracker for porting Nethermind tests

[LouisTsai-Csie]

Summary

The Nethermind team is currently working on benchmark tests to support raising the gas limit. This issue serves as a tracker for porting tests from this PR.

If the test case already exists in EEST, there’s no need to port it from Nethermind, please provide a link to the relevant EEST code snippet. If the test needs to be added, kindly share the PR link.

Category

block

• Warmup
  • Warmup - Done, Discussion
• ETH transfers: Link
  • ETH transfers - feat(benchmark): create pure ether transfer worst case #1742
• Transactions: Link
  • Tx with big zero data - feat(benchmark): add pure calldata transaction case #1820

Opcode

Arithmetic

Cryptographic

• KECCAK256 (0x20): Link - Done, Discussion
  • Keccak256 from 1 byte
  • Keccak256 from 8 bytes
  • Keccak256 from 32 bytes

Stateful Opcodes

• SELFBALANCE (0x47) - refactor(tests): update MSIZE and SELFBALANCE benchmark test #1771
  • SimpleInstructionTwoContracts (Endlessly pushing self balance to stack (1000 per 1 contract))

Termination

Environment

• CALLER (0x33)
  • SimpleInstructionTwoContracts (Endlessly pushing caller address to stack (1000 per 1 contract)) - refactor(benchmark): optimize the zero param opcode case #1737
  • SimpleInstructionSingleContract (Endlessly pushing caller address to stack, then popping it) -> Discussion
• ADDRESS (0x30)
  • SimpleInstructionTwoContracts (Endlessly pushing account address to stack (1000 per 1 contract)) - refactor(benchmark): optimize the zero param opcode case #1737
• ORIGIN (0x32)
  • SimpleInstructionTwoContracts (Endlessly pushing execution origination address to stack (1000 per 1 contract)) - refactor(benchmark): optimize the zero param opcode case #1737
• TIMESTAMP (0x42)
  • SimpleInstructionTwoContracts (Endlessly pushing current block's timestamp to stack (1000 per 1 contract)) - refactor(benchmark): optimize the zero param opcode case #1737
• NUMBER (0x43)
  • SimpleInstructionTwoContracts (Endlessly pushing current block's number to stack (1000 per 1 contract)) - refactor(benchmark): optimize the zero param opcode case #1737
• PREVRANDAO (0x44)
  • SimpleInstructionTwoContracts (Endlessly pushing previous block's randao mix to stack (1000 per 1 contract)) - refactor(benchmark): optimize the zero param opcode case #1737
• GASLIMIT (0x45)
  • SimpleInstructionTwoContracts (Endlessly pushing current block's gas limit to stack (1000 per 1 contract)) - refactor(benchmark): optimize the zero param opcode case #1737
• CHAINID (0x46)
  • SimpleInstructionTwoContracts (Endlessly pushing chain ID to stack (1000 per 1 contract)) - refactor(benchmark): optimize the zero param opcode case #1737
• BASEFEE (0x48)
  • SimpleInstructionTwoContracts (Endlessly pushing current base fee to stack (1000 per 1 contract)) - refactor(benchmark): optimize the zero param opcode case #1737
• CODECOPY (0x39): Link
  • CODECOPY - test(benchmark): add coverage for CALLDATACOPY, CODECOPY, RETURNDATACOPY and MCOPY #1800

Block & Chain Info

• COINBASE (0x41)
  • SimpleInstructionTwoContracts (Endlessly pushing current block's coinbase to stack (1000 per 1 contract)) - refactor(benchmark): optimize the zero param opcode case #1737

Blobs

• BLOBBASEFEE - refactor(benchmark): optimize the zero param opcode case #1737
  • SimpleInstructionTwoContracts (Endlessly pushing current blob base fee to stack (1000 per 1 contract))
• BLOBHASH
  • SimpleInstructionTwoContracts (Endlessly pushing zero as index and BlobHash opcode to stack when there were no blobs (1000 per 1 contract)) - refactor(benchmark): refactor blobhash opcode worst case #1783

Stack & Memory

• PUSH0 (0x5f)
  • SimpleInstructionSingleContract (Endlessly pushing zeros to stack, then popping it) - Discussion
  • SimpleInstructionTwoContracts (Endlessly pushing zeros to stack (1000 per 1 contract)) - feat(tests): add worst case tests for PUSH opcodes #1762
• GAS (0x5a)
  • SimpleInstructionSingleContract (Endlessly pushing amount of remaining gas to stack (1000 per 1 contract)) -> Discussion
  • SimpleInstructionTwoContracts (Endlessly pushing amount of remaining gas to stack, then popping it) - refactor(benchmark): optimize the zero param opcode case #1737
• MSIZE (0x59) - refactor(tests): update MSIZE and SELFBALANCE benchmark test #1771
  • MSize
• MSTORE (0x52): Link - Done, Discussion
  • MStore - zero
  • MStore - random
• TSTORE (0x5c) - Done, Discussion
  • TStore - one storage key, repeating zero value
  • TStore - one storage key, repeating constant value
  • TStore - one storage key, repeating random values
• SSTORE (0x55): Link1, Link2, Link3.
  • SStore - one storage key, repeating zero value - Refactor(benchmark): optimize storage access worst case #1813
  • SStore - one storage key, repeating constant value - Refactor(benchmark): optimize storage access worst case #1813
  • SStore - one storage key, repeating random values - Refactor(benchmark): optimize storage access worst case #1813
  • SStore - one storage key, repeating two values, zero and non-zero - Refactor(benchmark): optimize storage access worst case #1813
  • SStore - many accounts, consecutive storage keys, random values - Refactor(benchmark): optimize storage access worst case #1813
  • SStore - many accounts, random storage keys, random values - Refactor(benchmark): optimize storage access worst case #1813
  • SStore - many accounts, consecutive storage keys, zero values - Refactor(benchmark): optimize storage access worst case #1813
  • SStore - many accounts, random storage keys, zero values - Refactor(benchmark): optimize storage access worst case #1813

Control Flow

• JUMPDEST (0x5b)
  • SimpleInstructionSingleContract (Block full of JumpDest opcode only) -> refactor: optimize jumpdest test case #1732

Precompiled Contracts

• EcRecover (0x01): Link
  • EcRecover precompile -> Done, Discussion
• SHA-2 (0x02) - Done, Discussion
  • SHA-2 precompile from 1 byte
  • SHA-2 precompile from 8 bytes
  • SHA-2 precompile from 32 bytes
  • SHA-2 precompile from 128 bytes
  • SHA-2 precompile from 1024 bytes
  • SHA-2 precompile from 16k bytes
• Ripemd-160 (0x03) - Done, Discussion
  • Ripemd-160 precompile from 1 byte
  • Ripemd-160 precompile from 8 bytes
  • Ripemd-160 precompile from 32 bytes
  • Ripemd-160 precompile from 128 bytes
  • Ripemd-160 precompile from 1024 bytes
  • Ripemd-160 precompile from 16k bytes
• Identity (0x04) - Done, Discussion
  • Identity precompile from 1 byte
  • Identity precompile from 8 bytes
  • Identity precompile from 32 bytes
  • Identity precompile from 128 bytes
  • Identity precompile from 1024 bytes
  • Identity precompile from 16k bytes
• Modexp (0x05): Link1, Link2
  • Modexp min gas, base heavy - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp min gas, exp heavy - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp min gas, balanced - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp 208 gas, balanced - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp 215 gas, exp heavy - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp 298 gas, exp heavy - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp Pawel 2 - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp Pawel 3 - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp Pawel 4 - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp 408 gas, base heavy - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp 400 gas, exp heavy - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp 408 gas, balanced - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp 616 gas, base heavy - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp 600 gas, exp heavy - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp 600 gas, balanced - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp 800 gas, base heavy - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp 800 gas, exp heavy - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp 767 gas, balanced - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp 852 gas, exp heavy - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp 867 gas, base heavy - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp 996 gas, balanced - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp 1045 gas, base heavy - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp 677 gas, balanced - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp 765 gas, balanced - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp 1360 gas, balanced - test(benchmark): port nethermind modexp benchmark case #1802
  • Modexp "eip_example1" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "eip_example2" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "nagydani-1-square" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "nagydani-1-qube" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "nagydani-1-pow0x10001" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "nagydani-2-square" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "nagydani-2-qube" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "nagydani-2-pow0x10001" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "nagydani-3-square" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "nagydani-3-qube" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "nagydani-3-pow0x10001" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "nagydani-4-square" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "nagydani-4-qube" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "nagydani-4-pow0x10001" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "nagydani-5-square" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "nagydani-5-qube" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "nagydani-5-pow0x10001" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "marius-1-even" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "guido-1-even" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "guido-2-even" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "guido-3-even" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "guido-4-even"
  • Modexp "pawel-1-exp-heavy" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "pawel-2-exp-heavy" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "pawel-3-exp-heavy" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp "pawel-4-exp-heavy" - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp common 1360 1 - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp common 1360 2 - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp common 1349 1 - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp common 1152 1 - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp common 200 1 - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp common 200 2 - tests(benchmark): expand worst case modexp benchmarks #1780
  • Modexp common 200 3 - tests(benchmark): expand worst case modexp benchmarks #1780
• EcAdd (0x06): Link
  • EcAdd with (0, 0) - test(benchmark): add extra test case for ecAdd, ecMul and ecPairing #1799
  • EcAdd with (1, 2) - test(benchmark): add extra test case for ecAdd, ecMul and ecPairing #1799
  • EcAdd with 32-byte coordinates - test(benchmark): add extra test case for ecAdd, ecMul and ecPairing #1799
• EcMul (0x07): Link
  • EcMul with (0, 0) and scalar 2 - test(benchmark): add extra test case for ecAdd, ecMul and ecPairing #1799
  • EcMul with (0, 0) and 32-byte scalar - test(benchmark): add extra test case for ecAdd, ecMul and ecPairing #1799
  • EcMul with (1, 2) and scalar 2 - test(benchmark): add extra test case for ecAdd, ecMul and ecPairing #1799
  • EcMul with (1, 2) and 32-byte scalar - test(benchmark): add extra test case for ecAdd, ecMul and ecPairing #1799
  • EcMul with 32-byte coordinates and scalar 2 - test(benchmark): add extra test case for ecAdd, ecMul and ecPairing #1799
  • EcMul with 32-byte coordinates and 32-byte scalar - test(benchmark): add extra test case for ecAdd, ecMul and ecPairing #1799
• EcPairing (0x08): Link
  • EcPairing with empty input - test(benchmark): add extra test case for ecAdd, ecMul and ecPairing #1799
  • EcPairing with 2 sets of data - test(benchmark): add extra test case for ecAdd, ecMul and ecPairing #1799
• Blake2f (0x09): Link
  • Blake2f 1 round
  • Blake2f 1k rounds
  • Blake2f 1M rounds
  • Blake2f 10M rounds
• Point Evaluation (0x0a): Link - Done, Discussion
  • Point evaluation - one data
• Secp256r1 (0x0b): Link
  • Secp256r1 precompile, valid signature - feat(benchmark): add p256verify to worst compute tests #1744

Performance

Please start with the operation that has the lowest performance.

• besu - EcAddMarius1: 27.8
• besu - EmptyPayload: 28.8
• besu - EcAdd32ByteCoordinatesCACHABLE: 30.6
• besu - EcAdd12CACHABLE: 31.3
• reth - EmptyPayload: 33.4
• besu - Number: 33.4
• nethermind - EmptyPayload: 35.6
• reth - EcAddMarius1: 37.7
• besu - Keccak256From32Bytes: 38.5
• besu - EcRecoverUNCACHABLE: 40.0
• besu - Keccak256From8Bytes: 41.8
• nethermind - EcAddMarius1: 42.1
• besu - EcRecoverUNCACHABLE2: 45.3
• besu - Keccak256From1Byte: 46.2
• besu - Blake1KRoundsCACHABLE: 47.0
• besu - EcRecoverCACHABLE: 47.2
• besu - GasLimit: 49.8
• besu - Timestamp: 51.7
• besu - Origin: 52.4
• besu - Blake1MRoundsCACHABLE: 54.3
• besu - Gas: 56.0
• nethermind - EcRecoverUNCACHABLE: 56.7
• geth - EcRecoverUNCACHABLE: 56.9
• besu - MSize: 58.1
• nethermind - EcRecoverUNCACHABLE2: 58.4
• besu - Caller: 59.2
• besu - IdentityFrom1ByteCACHABLE: 59.6
• besu - PrevRandao: 59.9

[LouisTsai-Csie]

For opcodes that take zero inputs but push one element onto the stack, such as CALLER, ADDRESS, ORIGIN, and others. Nethermind uses two different testing approaches:

1. Op + POP Pattern: Repeatedly execute the opcode followed by a POP. Since each opcode pushes one item onto the stack, a subsequent POP prevents stack overflow (the EVM stack limit is 1024).
2. Two-Contract Pattern: Create one contract that executes the opcode 1024 times (filling the EVM stack), and another contract that repeatedly invokes the first contract using STATICCALL.

In this issue description, the first method is referred to as SimpleInstructionSingleContract, and the second as SimpleInstructionTwoContracts.

Our original test case, introduced in PR #1698, used the first method. However, after analyzing gas consumption, we found that the second method is more gas-efficient and can execute more cycles. Therefore, PR #1737 focuses on this optimization, with further details available in its description.

Summary:
We will not port the SimpleInstructionSingleContract scenario, as SimpleInstructionTwoContracts is more efficient and allows for more cycles. For full details and analysis, refer to PR #1737.

[LouisTsai-Csie]

ECRECOVER

Nethermind Version - Link

1. Push all necessary data (v, r, s fields) onto the stack, then store them in memory.
2. Repeat a code sequence that calls the ECRECOVER precompile 1000 times.
3. Jump back to the beginning of step 2 and repeat.

EEST Version - Link

This version follows a similar structure but includes optimizations. In step 2, the repeated code sequence is expanded as much as possible, up to the maximum contract code size limit. Step 3 remains the same as in the Nethermind version.

It’s unclear the original intention was in Nethermind’s version, but there is a counter used within the loop in step 2, a PUSH1 + ADD pattern, which adds extra gas cost per iteration.

Summary
Given the analysis above, it is not necessary to port the Nethermind test case for ecrecover. The existing EEST version already covers the scenario in a more gas-efficient way.

[LouisTsai-Csie]

Keccak256

Nethermind Version – Link

1. Push the CALLER onto the stack (this will be stored in memory later), followed by a JUMPDEST instruction to enable looping back and re-executing the code.
2. In each iteration, store the data into memory using MSTORE, then perform the KECCAK256 operation on a specified input size. The input size varies between 1, 8, or 32 bytes.
3. At the end of the sequence, jump back to the beginning to start the loop again.

EEST Version - Link

The KECCAK256 algorithm in Ethereum includes an internal permutation step, which is executed for every 136 bytes of input. In the EEST benchmark test, the goal is to maximize the number of permutations, not just the number of KECCAK256 calls.

To achieve this, the test:

• Calculates the available gas by subtracting the intrinsic gas cost and memory expansion cost from the block gas limit.
• Determines the gas cost of a single iteration (including the KECCAK256 call and related operations).
• Computes how many iterations fit within the available gas.
• Multiplies the number of iterations by the number of permutations per iteration (based on input size), and keeps track of the maximum.

Finally, it constructs a loop structure that repeatedly executes the KECCAK256 operation until all available gas is consumed.

Summary

This test will not be ported from Nethermind, as testing for maximum permutations is likely more effective. Further analysis can be found here.

[LouisTsai-Csie]

MSTORE

Nethermind Version – Link

There are two test cases for MSTORE: MStoreZero and MStoreRandom. The difference lies in the value being stored in memory,MStoreZero stores zero, while MStoreRandom pushes and stores a random value.

The test repeatedly executes this operation until reaching the gas limit.

EEST Version – Link

The EEST version follows a similar approach. It includes memory expansion costs and also repeatedly performs the MSTORE operation.

Summary

The MSTORE test in Nethermind is similar to the one in EEST. However, the EEST version covers more comprehensive conditions. As a result, the Nethermind version will not be ported.

[LouisTsai-Csie]

TSTORE

Nethermind Version – Link

There are three test cases: TStoreOneKeyZeroValue, TStoreOneKeyConstantValue, and TStoreOneKeyRandomValue.
The TSTORE operation takes two parameters: an offset and a value. In all cases, the offset is fixed at zero, while the value varies. The operation is repeated 500 times in each test.

EEST Version – Link

The EEST version introduces two variables to control different conditions: key_mut and dense_val_mut.

• If key_mut is True, the key is mutated on each iteration using Op.GAS, ensuring it changes every time.
• If dense_val_mut is True, GAS is also used as a cheap way to ensure each stored value differs from the previous one.

Additionally, the EEST test carefully considers the maximum code size to amortize the gas cost of JUMP operations, improving efficiency compared to the Nethermind version.

Summary

An equivalent test already exists in EEST with more advanced handling of gas and code structure, so the Nethermind version will not be ported.

[LouisTsai-Csie]

Warmup

Nethermind Version – Link 1, Link 2

The purpose of this test case is unclear, as it shares the same setup as the pure transfer test case, which is already covered here. It seems to be more as a “health check” to verify that the infrastructure is working correctly.

EEST Version

Pure Ether transfers are handled in PR #1742.
Warm and cold storage access cases are covered in the following tests:

• Line 49
• Line 139
• Line 211
• Line 317
• Line 450

Summary
Since the functionality is already covered and the intent of the warmup test is not clearly distinct, it will not be ported.

[LouisTsai-Csie]

SHA2-256, RIPEMD-160, Identity

Nethermind Version - Link

1. Six fixed payload sizes are tested: 1, 8, 32, 128, 1024, 16384 bytes.
2. Before every run a buffer of the desired length is filled with pseudo-random words: each 32-byte slot is the keccak256 hash of the previous one.
3. The buffer is copied to memory and the chosen precompile is invoked 100 times via STATICCALL, and then the context jumps back to the start of the code sequence and execute again until gas run out.

EEST Version - Link

The goal is not to fix the payload size but to maximise the number of compression-rounds (units of real computation) that fit into a single transaction gas-limit. The process works as follows:

1. Start with the block gas-limit, subtract the intrinsic transaction cost.
2. For every candidate input length (1 … 1 000 000 bytes, stepped by 32), compute the per-iteration gas usage, and the memory-expansion gas for the current length.
3. Calculate how many iterations count still fit in the gas available (Gas from step 1 - Memory expansion cost in step 2)
4. Convert input_length to “rounds of real work", by the bytes_per_unit_of_work and the number of iteration.
5. Find the input length that maximises rounds × iterations.
6. Repeatedly STATICCALLs the precompile with optimal input length.

Summary

The test coverage in Nethermind is more limited compared to EEST, as it relies on a set of fixed input sizes rather than dynamically optimizing for computational intensity.

In contrast, EEST explicitly searches for the input length that maximizes cryptographic computation under a given gas limit.
As a result, these three precompile tests (SHA-256, RIPEMD-160, and IDENTITY) will not be ported from Nethermind, since EEST already provides a more comprehensive test for each.

[LouisTsai-Csie]

Point Evalutation

Nethermind Version – Link

The Nethermind test includes only a single test case. It follows a straightforward approach: the bytes fields are written to memory, and the precompile is repeatedly invoked using STATICCALL.

EEST Version – Link

The EEST test adopts a similar structure, using the same inputs and repeating STATICCALL to evaluate the precompile’s behavior.

Summary

Since the EEST test already covers the same logic and structure, the Nethermind version will not be ported.