refactor(benchmark): rename zkevm tests to benchmark (#1804)

spencer-tb · danceratopz · marioevz · web-flow · commit d9521429b541 · 2025-07-01T21:33:21.000+02:00
* chore: rename zkevm tests to benchmark.

* chore(docs): changelog.

* refactor(tests,fw,tox,docs): rename zkevm marker to benchmark

* chore(tests): remove zkevm specific nomenclature from docstrings

* chore(ci|docs): rename zkevm releases to benchmark.

* chore(docs): align changelog docs for future releases.

* chore: final tweaks after rebase.

* Apply suggestions from code review

---------

Co-authored-by: danceratopz &lt;danceratopz@gmail.com&gt;
Co-authored-by: Mario Vega &lt;marioevz@gmail.com&gt;
diff --git a/.github/configs/evm.yaml b/.github/configs/evm.yaml
@@ -11,7 +11,7 @@ static:
   repo: ethereum/evmone
   ref: master
   targets: ["evmone-t8n", "evmone-eofparse"]
-zkevm:
+benchmark:
   impl: evmone
   repo: ethereum/evmone
   ref: master
diff --git a/.github/configs/feature.yaml b/.github/configs/feature.yaml
@@ -11,24 +11,24 @@ static:
   evm-type: static
   fill-params: --until=Prague --fill-static-tests ./tests/static
   solc: 0.8.21
-zkevm_36M:
-  evm-type: zkevm
-  fill-params: --from=Cancun --until=Prague --block-gas-limit 36000000 -m zkevm ./tests
+benchmark_30M:
+  evm-type: benchmark
+  fill-params: --from=Cancun --until=Prague --block-gas-limit 30000000 -m benchmark ./tests
   solc: 0.8.21
   feature_only: true
-zkevm_60M:
-  evm-type: zkevm
-  fill-params: --from=Cancun --until=Prague --block-gas-limit 60000000 -m zkevm ./tests
+benchmark_60M:
+  evm-type: benchmark
+  fill-params: --from=Cancun --until=Prague --block-gas-limit 60000000 -m benchmark ./tests
   solc: 0.8.21
   feature_only: true
-zkevm_90M:
-  evm-type: zkevm
-  fill-params: --from=Cancun --until=Prague --block-gas-limit 90000000 -m zkevm ./tests
+benchmark_90M:
+  evm-type: benchmark
+  fill-params: --from=Cancun --until=Prague --block-gas-limit 90000000 -m benchmark ./tests
   solc: 0.8.21
   feature_only: true
-zkevm_120M:
-  evm-type: zkevm
-  fill-params: --from=Cancun --until=Prague --block-gas-limit 120000000 -m zkevm ./tests
+benchmark_120M:
+  evm-type: benchmark
+  fill-params: --from=Cancun --until=Prague --block-gas-limit 120000000 -m benchmark ./tests
   solc: 0.8.21
   feature_only: true
 eip7692:
diff --git a/.github/workflows/tox_verify.yaml b/.github/workflows/tox_verify.yaml
@@ -182,8 +182,8 @@ jobs:
         env:
           EELS_RESOLUTIONS_FILE: ${{ github.workspace }}/.github/configs/eels_resolutions.json
         run: uvx --with=tox-uv tox -e tests-deployed
-  tests_deployed_zkevm:
-    name: Fill zkEVM test cases, deployed, ${{ matrix.os }}, ${{ matrix.python }}
+  tests_deployed_benchmark:
+    name: Fill benchmark test cases, deployed, ${{ matrix.os }}, ${{ matrix.python }}
     runs-on: ${{ matrix.os }}
     strategy:
       matrix:
@@ -209,4 +209,4 @@ jobs:
       - name: Run tox - fill tests for deployed forks
         env:
           EELS_RESOLUTIONS_FILE: ${{ github.workspace }}/.github/configs/eels_resolutions.json
-        run: uvx --with=tox-uv tox -e tests-deployed-zkevm
+        run: uvx --with=tox-uv tox -e tests-deployed-benchmark
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
@@ -17,9 +17,11 @@ The output behavior of `fill` has changed ([#1608](https://github.com/ethereum/e
 - Before: `fill` wrote fixtures into the directory specified by the `--output` flag (default: `fixtures`). This could have many unintended consequences, including unexpected errors if old or invalid fixtures existed in the directory (for details see [#1030](https://github.com/ethereum/execution-spec-tests/issues/1030)).
 - Now: `fill` will exit without filling any tests if the specified directory exists and is not-empty. This may be overridden by adding the `--clean` flag, which will first remove the specified directory.
 
-#### Feature `zkevm` expanded to `zkevm_36M`, `zkevm_60M`, `zkevm_90M`, `zkevm_120M`
+#### Feature `zkevm` updated to `benchmark`
 
-Renames `zkevm` to `zkevm_36M` and further expands the zkEVM features to run using 60M, 90M and 120M block gas limits in `fixtures_zkevm_36M.tar.gz`, `fixtures_zkevm_60M.tar.gz`, `fixtures_zkevm_90M.tar.gz` and `fixtures_zkevm_120M.tar.gz` respectively.
+Due to the crossover between `zkevm` and `benchmark` tests, all instances of the former have been replaced with the latter nomenclature. Repository PR labels and titles are additionally updated to reflect this change.
+
+This update renames the `zkevm` feature release to `benchmark_30M` and further expands the latter for 60M, 90M, and 120M block gas limits in `fixtures_benchmark_30M.tar.gz`, `fixtures_benchmark_60M.tar.gz`, `fixtures_benchmark_90M.tar.gz`, and `fixtures_benchmark_120M.tar.gz` respectively.
 
 Users can select any of the artifacts depending on their testing needs for their provers.
 
@@ -38,7 +40,7 @@ Users can select any of the artifacts depending on their testing needs for their
 - 🔀 Refactor: Encapsulate `fill`'s fixture output options (`--output`, `--flat-output`, `--single-fixture-per-file`) into a `FixtureOutput` class ([#1471](https://github.com/ethereum/execution-spec-tests/pull/1471),[#1612](https://github.com/ethereum/execution-spec-tests/pull/1612)).
 - ✨ Don't warn about a "high Transaction gas_limit" for `zkevm` tests ([#1598](https://github.com/ethereum/execution-spec-tests/pull/1598)).
 - 🐞 `fill` no longer writes generated fixtures into an existing, non-empty output directory; it must now be empty or `--clean` must be used to delete it first ([#1608](https://github.com/ethereum/execution-spec-tests/pull/1608)).
-- 🐞 zkEVM marked tests have been removed from `tests-deployed` tox environment into its own separate workflow `tests-deployed-zkevm` and are filled by `evmone-t8n` ([#1617](https://github.com/ethereum/execution-spec-tests/pull/1617)).
+- 🐞 `zkevm` marked tests have been removed from `tests-deployed` tox environment into its own separate workflow `tests-deployed-zkevm` and are filled by `evmone-t8n` ([#1617](https://github.com/ethereum/execution-spec-tests/pull/1617)).
 - ✨ Field `postStateHash` is now added to all `blockchain_test` and `blockchain_test_engine` tests that use `exclude_full_post_state_in_output` in place of `postState`. Fixes `evmone-blockchaintest` test consumption and indirectly fixes coverage runs for these tests ([#1667](https://github.com/ethereum/execution-spec-tests/pull/1667)).
 - 🔀 Changed INVALID_DEPOSIT_EVENT_LAYOUT to a BlockException instead of a TransactionException ([#1773](https://github.com/ethereum/execution-spec-tests/pull/1773)).
 
@@ -72,6 +74,7 @@ Users can select any of the artifacts depending on their testing needs for their
 - ✨ Added automatic checklist generation for every EIP inside of the `tests` folder. The checklist is appended to each EIP in the documentation in the "Test Case Reference" section ([#1679](https://github.com/ethereum/execution-spec-tests/pull/1679), [#1718](https://github.com/ethereum/execution-spec-tests/pull/1718)).
 - 🔀 Add macOS hive development mode workaround to the docs [#1786](https://github.com/ethereum/execution-spec-tests/pull/1786).
 - 🔀 Refactor and clean up of exceptions including EOF exceptions within client specific mappers [#1803](https://github.com/ethereum/execution-spec-tests/pull/1803).
+- 🔀 Rename `tests/zkevm/` to `tests/benchmark/` and replace the `zkevm` pytest mark with `benchmark` [#1804](https://github.com/ethereum/execution-spec-tests/pull/1804).
 
 ### 🧪 Test Cases
 
@@ -82,7 +85,7 @@ Users can select any of the artifacts depending on their testing needs for their
 - ✨ EIP-7823, EIP-7883: Add test cases for ModExp precompile gas-cost updates and input limits on Osaka ([#1579](https://github.com/ethereum/execution-spec-tests/pull/1579), [#1729](https://github.com/ethereum/execution-spec-tests/pull/1729)).
 - ✨ [EIP-7825](https://eips.ethereum.org/EIPS/eip-7825): Add test cases for the transaction gas limit of 30M gas ([#1711](https://github.com/ethereum/execution-spec-tests/pull/1711)).
 - ✨ [EIP-7951](https://eips.ethereum.org/EIPS/eip-7951): add test cases for `P256VERIFY` precompile to support secp256r1 curve [#1670](https://github.com/ethereum/execution-spec-tests/pull/1670).
-- ✨ Introduce blockchain tests for ZKEVM to cover the scenario of pure ether transfers [#1742](https://github.com/ethereum/execution-spec-tests/pull/1742).
+- ✨ Introduce blockchain tests for benchmark to cover the scenario of pure ether transfers [#1742](https://github.com/ethereum/execution-spec-tests/pull/1742).
 - ✨ [EIP-7934](https://eips.ethereum.org/EIPS/eip-7934): Add test cases for the block RLP max limit of 10MiB ([#1730](https://github.com/ethereum/execution-spec-tests/pull/1730)).
 - ✨ [EIP-7939](https://eips.ethereum.org/EIPS/eip-7939) Add count leading zeros (CLZ) opcode tests for Osaka ([#1733](https://github.com/ethereum/execution-spec-tests/pull/1733)).
 - ✨ [EIP-7918](https://eips.ethereum.org/EIPS/eip-7918): Blob base fee bounded by execution cost test cases (initial), includes some adjustments to [EIP-4844](https://eips.ethereum.org/EIPS/eip-4844) tests ([#1685](https://github.com/ethereum/execution-spec-tests/pull/1685)).
@@ -109,12 +112,12 @@ A new fork `EOFv1` has additionally been created to fill and consume EOF related
 - ✨ Engine API updates for Osaka, add `get_blobs` rpc method ([#1510](https://github.com/ethereum/execution-spec-tests/pull/1510)).
 - ✨ The EIP Version checker has been moved from `fill` and `execute` to it's own command-line tool `check_eip_versions` that gets ran daily as a Github Action ([#1537](https://github.com/ethereum/execution-spec-tests/pull/1537)).
 - 🔀 Add new `tests/unscheduled` folder, move EOF from Osaka to unscheduled, add `EOFv1` fork name for EOF tests ([#1507](https://github.com/ethereum/execution-spec-tests/pull/1507)).
-- ✨ CI features now contain an optional field to skip them from EEST full releases, `zkevm` and EOF features are now feature only ([#1596](https://github.com/ethereum/execution-spec-tests/pull/1596)).
+- ✨ CI features now contain an optional field to skip them from EEST full releases, `benchmark` and EOF features are now feature only ([#1596](https://github.com/ethereum/execution-spec-tests/pull/1596)).
 - 🐞 Don't attempt to install `solc` via `solc-select` on ARM (official Linux ARM builds of `solc` are not available at the time of writing, cf [ethereum/solidity#11351](https://github.com/ethereum/solidity/issues/11351)) and add a version sanity check ([#1556](https://github.com/ethereum/execution-spec-tests/pull/1556)).
 
 ### 🧪 Test Cases
 
-- 🔀 Automatically apply the `zkevm` marker to all tests under `./tests/zkevm/` and `./tests/prague/eip2537_bls_12_381_precompiles/` via conftest configuration ([#1534](https://github.com/ethereum/execution-spec-tests/pull/1534)).
+- 🔀 Automatically apply the `benchmark` marker to all tests under `./tests/benchmark/` and `./tests/prague/eip2537_bls_12_381_precompiles/` via conftest configuration ([#1534](https://github.com/ethereum/execution-spec-tests/pull/1534)).
 - ✨ Port [calldataload](https://github.com/ethereum/tests/blob/ae4791077e8fcf716136e70fe8392f1a1f1495fb/src/GeneralStateTestsFiller/VMTests/vmTests/calldatacopyFiller.yml) and [calldatasize](https://github.com/ethereum/tests/blob/81862e4848585a438d64f911a19b3825f0f4cd95/src/GeneralStateTestsFiller/VMTests/vmTests/calldatasizeFiller.yml) tests ([#1236](https://github.com/ethereum/execution-spec-tests/pull/1236)).
 
 ## [v4.4.0](https://github.com/ethereum/execution-spec-tests/releases/tag/v4.4.0) - 2025-04-29
@@ -131,9 +134,9 @@ The tests have been filled using the new static test filler introduced in [#1336
 
 Users can expect that all tests currently living in [ethereum/tests](https://github.com/ethereum/tests/tree/develop/src) should eventually make its way into [`./tests/static`](https://github.com/ethereum/execution-spec-tests/tree/main/tests/static) and can rely that these tests, filled for new forks even, will be included in `fixtures_static.tar.gz`.
 
-#### `fixtures_zkevm`
+#### `fixtures_benchmark`
 
-Another new fixture tarball has been included in this release: `fixtures_zkevm.tar.gz`.
+Another new fixture tarball has been included in this release: `fixtures_benchmark.tar.gz`.
 
 Includes tests that are tailored specifically to test the execution layer proof generators.
 
diff --git a/docs/getting_started/code_standards.md b/docs/getting_started/code_standards.md
@@ -12,7 +12,7 @@ Code pushed to @ethereum/execution-spec-tests must fulfill the following checks
 | Typecheck              | `uvx --with=tox-uv tox -e typecheck`            | Objects that provide typehints pass type-checking via `mypy`.                                               |
 | Framework unit tests   | `uvx --with=tox-uv tox -e pytest`               | All framework unit tests must execute correctly.                                                            |
 | EL Client test cases   | `uvx --with=tox-uv tox -e tests-deployed`       | All client test cases for deployed forks can be generated.                                                  |
-| zkEVM EL Test cases    | `uvx --with=tox-uv tox -e tests-deployed-zkevm` | All client test cases specific to zkEVMs for deployed forks can be generated.                               |
+| Benchmark EL Test cases    | `uvx --with=tox-uv tox -e tests-deployed-benchmark` | All client test cases specific to benchmarks for deployed forks can be generated.                               |
 | HTML doc build         | `uvx --with=tox-uv tox -e mkdocs`               | Documentation generated without warnings.                                                                   |
 | Spellcheck             | `uvx --with=tox-uv tox -e spellcheck`           | Code and documentation spell-check using codespell. |
 | Markdown lint          | `uvx --with=tox-uv tox -e markdownlint`         | Markdown lint (requires [additional dependency](code_standards_details.md#additional-dependencies)).        |
diff --git a/docs/running_tests/releases.md b/docs/running_tests/releases.md
@@ -46,7 +46,7 @@ These releases are tagged using the format `<pre_release_name>@vX.Y.Z`.
 Examples:
 
 - [`fusaka-devnet-1@v1.0.0`](https://github.com/ethereum/execution-spec-tests/releases/tag/fusaka-devnet-1%40v1.0.0) - this fixture release contains tests adhering to the [Fusaka Devnet 1 spec](https://notes.ethereum.org/@ethpandaops/fusaka-devnet-1).
-- [`zkevm@v1.0.0`](https://github.com/ethereum/execution-spec-tests/releases/tag/zkevm%40v0.1.0) - this fixture release contains tests specifically aimed at testing zkEVMs.
+- [`benchmark@v1.0.0`](https://github.com/ethereum/execution-spec-tests/releases/tag/benchmark%40v1.0.0) - this fixture release contains tests specifically aimed at benchmarking EVMs.
 
 Devnet releases should be treated as WIP and may not yet contain full test coverage (or even coverage for all EIPs). The coverage provided by these releases is detailed in the corresponding release notes.
 
diff --git a/src/ethereum_test_specs/base.py b/src/ethereum_test_specs/base.py
@@ -161,8 +161,8 @@ def is_tx_gas_heavy_test(self) -> bool:
         if self._request is not None and hasattr(self._request, "node"):
             node = self._request.node
             has_slow_marker = node.get_closest_marker("slow") is not None
-            has_zkevm_marker = node.get_closest_marker("zkevm") is not None
-            return has_slow_marker or has_zkevm_marker
+            has_benchmark_marker = node.get_closest_marker("benchmark") is not None
+            return has_slow_marker or has_benchmark_marker
         return False
 
     def is_exception_test(self) -> bool | None:
diff --git a/src/pytest_plugins/shared/execute_fill.py b/src/pytest_plugins/shared/execute_fill.py
@@ -83,7 +83,7 @@ def pytest_configure(config: pytest.Config):
     )
     config.addinivalue_line(
         "markers",
-        "zkevm: Tests that are relevant to zkEVM.",
+        "benchmark: Tests relevant to benchmarking EVMs.",
     )
     config.addinivalue_line(
         "markers",
diff --git a/tests/benchmark/__init__.py b/tests/benchmark/__init__.py
diff --git a/tests/benchmark/conftest.py b/tests/benchmark/conftest.py
@@ -1,12 +1,12 @@
-"""Pytest configuration for zkEVM tests."""
+"""Pytest configuration for benchmark tests."""
 
 from pathlib import Path
 
 import pytest
 
 
 def pytest_collection_modifyitems(config, items):
-    """Add the `zkevm` marker to all tests under `./tests/zkevm`."""
+    """Add the `benchmark` marker to all tests under `./tests/benchmark`."""
     for item in items:
         if Path(__file__).parent in Path(item.fspath).parents:
-            item.add_marker(pytest.mark.zkevm)
+            item.add_marker(pytest.mark.benchmark)
diff --git a/tests/benchmark/helpers.py b/tests/benchmark/helpers.py
diff --git a/tests/benchmark/test_worst_blocks.py b/tests/benchmark/test_worst_blocks.py
@@ -1,8 +1,8 @@
 """
-abstract: Tests zkEVMs worst-case block scenarios.
-    Tests zkEVMs worst-case block scenarios.
+abstract: Tests that benchmark EVMs in worst-case block scenarios.
+    Tests that benchmark EVMs in worst-case block scenarios.
 
-Tests running worst-case block scenarios for zkEVMs.
+Tests running worst-case block scenarios for EVMs.
 """
 
 import pytest
diff --git a/tests/benchmark/test_worst_bytecode.py b/tests/benchmark/test_worst_bytecode.py
@@ -1,8 +1,8 @@
 """
-abstract: Tests for zkEVMs
-    Tests for zkEVMs.
+abstract: Tests that benchmark EVMs in worst-case opcode scenarios.
+    Tests that benchmark EVMs in worst-case opcode scenarios.
 
-Tests for zkEVMs worst-cases scenarios.
+Tests that benchmark EVMs in worst-case opcode scenarios.
 """
 
 import math
diff --git a/tests/benchmark/test_worst_compute.py b/tests/benchmark/test_worst_compute.py
@@ -1,8 +1,8 @@
 """
-abstract: Tests zkEVMs worst-case compute scenarios.
-    Tests zkEVMs worst-case compute scenarios.
+abstract: Tests that benchmark EVMs in worst-case compute scenarios.
+    Tests that benchmark EVMs in worst-case compute scenarios.
 
-Tests running worst-case compute opcodes and precompile scenarios for zkEVMs.
+Tests that benchmark EVMs when running worst-case compute opcodes and precompile scenarios.
 """
 
 import math
@@ -1136,7 +1136,6 @@ def test_worst_precompile_fixed_cost(
     )
 
 
-@pytest.mark.zkevm
 @pytest.mark.valid_from("Cancun")
 @pytest.mark.slow
 def test_worst_jumps(state_test: StateTestFiller, pre: Alloc):
@@ -1160,7 +1159,6 @@ def test_worst_jumps(state_test: StateTestFiller, pre: Alloc):
     )
 
 
-@pytest.mark.zkevm
 @pytest.mark.valid_from("Cancun")
 @pytest.mark.slow
 def test_worst_jumpdests(state_test: StateTestFiller, pre: Alloc, fork: Fork):
diff --git a/tests/benchmark/test_worst_opcode.py b/tests/benchmark/test_worst_opcode.py
@@ -1,8 +1,8 @@
 """
-abstract: Tests zkEVMs worst-case opcode scenarios.
-    Tests zkEVMs worst-case opcode scenarios.
+abstract: Tests benchmark worst-case opcode scenarios.
+    Tests benchmark worst-case opcode scenarios.
 
-Tests running worst-case opcodes scenarios for zkEVMs.
+Tests running worst-case opcodes scenarios for benchmarking purposes.
 """
 
 import pytest
diff --git a/tests/benchmark/test_worst_stateful_opcodes.py b/tests/benchmark/test_worst_stateful_opcodes.py
@@ -1,8 +1,8 @@
 """
-abstract: Tests zkEVMs worst-case stateful opcodes.
-    Tests zkEVMs worst-case stateful opcodes.
+abstract: Tests that benchmark EVMs for worst-case stateful opcodes.
+    Tests that benchmark EVMs for worst-case stateful opcodes.
 
-Tests running worst-case stateful opcodes for zkEVMs.
+Tests that benchmark EVMs for worst-case stateful opcodes.
 """
 
 import math
diff --git a/tox.ini b/tox.ini
diff --git a/whitelist.txt b/whitelist.txt

Original file line number	Diff line number	Diff line change
`@@ -83,7 +83,7 @@ def pytest_configure(config: pytest.Config):`
`83`	`83`	`)`
`84`	`84`	`config.addinivalue_line(`
`85`	`85`	`"markers",`
`86`		`- "zkevm: Tests that are relevant to zkEVM.",`
	`86`	`+ "benchmark: Tests relevant to benchmarking EVMs.",`
`87`	`87`	`)`
`88`	`88`	`config.addinivalue_line(`
`89`	`89`	`"markers",`