feat(plugins,tests): Add flag to configure max_gas for fill command (ethereum#1470)

* Add flag to configure `max_gas` for fill command

Add fixture for `max_gas` with default of 30_000_000

* Warnings for blockchain and state tests using gas_limit >= 30_000_000

* Set gas_limit from fixture for fill command

* Update CHANGELOG

Update fill command output in docs

* Add env fixture to pytest fill plugin and pass Environment to state tests directly

* Updates for using EnvironmentDefaults

* refactor(plugins/shared): Move block gas limit flag to shared

* refactor(tests): test fixes

* fix(specs): Unit tests

* fix(specs): More unit tests

* fix(types): Change semantics of max block gas limit

* refactor(plugins): Use different max gas defaults for execute/fill

* revert(pytest.ini): Remove filler.env

* fix(tests): Reduce amount of excessive-gas-usage tests

* fix(types): Fix unit tests take 3

* fix(tests): Mark tests timing out as slow

* Update docs for max gas flag

* fix(plugins): Move environment defaults modification in pytest_config

* fix tox

* Apply suggestions from code review

---------

Co-authored-by: Mario Vega <marioevz@gmail.com>

Author: 2 people

docs/CHANGELOG.md

@@ -81,6 +81,7 @@ This feature can be disabled by using `--disable-strict-exception-matching` for
 - ✨ The `static_filler` plug-in now has support for static state tests (from [GeneralStateTests](https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller)) ([#1362](https://github.com/ethereum/execution-spec-tests/pull/1362)).
 - ✨ Introduce `pytest.mark.exception_test` to mark tests that contain an invalid transaction or block ([#1436](https://github.com/ethereum/execution-spec-tests/pull/1436)).
 - 🐞 Fix `DeprecationWarning: Pickle, copy, and deepcopy support will be removed from itertools in Python 3.14.` by avoiding use `itertools` object in the spec `BaseTest` pydantic model ([#1414](https://github.com/ethereum/execution-spec-tests/pull/1414)).
+- ✨ An optional configuration flag to override the maximum gas limit in the environment for filling or executing tests is now available. The `--block-gas-limit` flag overrides the default block gas limit during filling. The `--transaction-gas-limit` flag overrides the maximum for transactions during execution. ([#1470](https://github.com/ethereum/execution-spec-tests/pull/1470)).
 
 #### `consume`
 

docs/filling_tests/filling_tests_command_line.md

@@ -115,82 +115,74 @@ Output:
 
 ```text
 usage: fill [-h] [--strict-alloc] [--ca-start CA_START] [--ca-incr CA_INCR]
-            [--evm-code-type EVM_CODE_TYPE] [--solc-bin SOLC_BIN]
-            [--solc-version SOLC_VERSION] [--evm-bin EVM_BIN] [--traces]
-            [--verify-fixtures] [--verify-fixtures-bin VERIFY_FIXTURES_BIN]
-            [--filler-path FILLER_PATH] [--output OUTPUT] [--flat-output]
-            [--single-fixture-per-file] [--no-html] [--strict-alloc]
-            [--ca-start CA_START] [--ca-incr CA_INCR] [--build-name BUILD_NAME]
-            [--evm-dump-dir EVM_DUMP_DIR] [--forks] [--fork FORK] [--from FROM]
-            [--until UNTIL]
+            [--evm-code-type EVM_CODE_TYPE] [--solc-bin SOLC_BIN] [--solc-version SOLC_VERSION]
+            [--evm-bin EVM_BIN] [--traces] [--verify-fixtures]
+            [--verify-fixtures-bin VERIFY_FIXTURES_BIN] [--filler-path FILLER_PATH] [--output OUTPUT]
+            [--flat-output] [--single-fixture-per-file] [--no-html] [--build-name BUILD_NAME]
+            [--skip-index SKIP_INDEX] [--block-gas-limit BLOCK_GAS_LIMIT] [--evm-dump-dir EVM_DUMP_DIR]
+            [--skip-evm-dump] [--forks] [--fork FORK] [--from FROM] [--until UNTIL]
 
 options:
   -h, --help            show this help message and exit
 
-Arguments defining pre-allocation behavior.:
-  --strict-alloc        [DEBUG ONLY] Disallows deploying a contract in a
-                        predefined address.
-  --ca-start CA_START, --contract-address-start CA_START
-                        The starting address from which tests will deploy
-                        contracts.
-  --ca-incr CA_INCR, --contract-address-increment CA_INCR
-                        The address increment value to each deployed contract by
-                        a test.
+Arguments defining pre-allocation behavior during test filling.:
+  --strict-alloc        [DEBUG ONLY] Disallows deploying a contract in a predefined address.
+  --ca-start, --contract-address-start CA_START
+                        The starting address from which tests will deploy contracts.
+  --ca-incr, --contract-address-increment CA_INCR
+                        The address increment value to each deployed contract by a test.
   --evm-code-type EVM_CODE_TYPE
                         Type of EVM code to deploy in each test by default.
 
 Arguments defining the solc executable:
-  --solc-bin SOLC_BIN   Path to a solc executable (for Yul source compilation).
-                        No default; if unspecified `--solc-version` is used.
+  --solc-bin SOLC_BIN   Path to a solc executable (for Yul source compilation). No default; if
+                        unspecified `--solc-version` is used.
   --solc-version SOLC_VERSION
                         Version of the solc compiler to use. Default: 0.8.24.
 
 Arguments defining evm executable behavior:
-  --evm-bin EVM_BIN     Path to an evm executable that provides `t8n`. Default:
-                        First 'evm' entry in PATH.
-  --traces              Collect traces of the execution information from the
-                        transition tool.
-  --verify-fixtures     Verify generated fixture JSON files using geth's evm
-                        blocktest command. By default, the same evm binary as for
-                        the t8n tool is used. A different (geth) evm binary may
-                        be specified via --verify-fixtures-bin, this must be
-                        specified if filling with a non-geth t8n tool that does
-                        not support blocktest.
+  --evm-bin EVM_BIN     Path to an evm executable (or name of an executable in the PATH) that provides
+                        `t8n`. Default: `ethereum-spec-evm-resolver`.
+  --traces              Collect traces of the execution information from the transition tool.
+  --verify-fixtures     Verify generated fixture JSON files using geth's evm blocktest command. By
+                        default, the same evm binary as for the t8n tool is used. A different (geth) evm
+                        binary may be specified via --verify-fixtures-bin, this must be specified if
+                        filling with a non-geth t8n tool that does not support blocktest.
   --verify-fixtures-bin VERIFY_FIXTURES_BIN
-                        Path to an evm executable that provides the `blocktest`
-                        command. Default: The first (geth) 'evm' entry in PATH.
+                        Path to an evm executable that provides the `blocktest` command. Default: The
+                        first (geth) 'evm' entry in PATH.
 
 Arguments defining filler location and output:
   --filler-path FILLER_PATH
                         Path to filler directives
-  --output OUTPUT       Directory path to store the generated test fixtures. If
-                        the specified path ends in '.tar.gz', then the specified
-                        tarball is additionally created (the fixtures are still
-                        written to the specified path without the '.tar.gz'
-                        suffix). Can be deleted. Default: './fixtures'.
-  --flat-output         Output each test case in the directory without the folder
-                        structure.
+  --output OUTPUT       Directory path to store the generated test fixtures. If the specified path ends
+                        in '.tar.gz', then the specified tarball is additionally created (the fixtures
+                        are still written to the specified path without the '.tar.gz' suffix). Can be
+                        deleted. Default: './fixtures'.
+  --flat-output         Output each test case in the directory without the folder structure.
   --single-fixture-per-file
-                        Don't group fixtures in JSON files by test function;
-                        write each fixture to its own file. This can be used to
-                        increase the granularity of --verify-fixtures.
-  --no-html             Don't generate an HTML test report (in the output
-                        directory). The --html flag can be used to specify a
-                        different path.
+                        Don't group fixtures in JSON files by test function; write each fixture to its
+                        own file. This can be used to increase the granularity of --verify-fixtures.
+  --no-html             Don't generate an HTML test report (in the output directory). The --html flag can
+                        be used to specify a different path.
   --build-name BUILD_NAME
-                        Specify a build name for the fixtures.ini file, e.g.,
-                        'stable'.
-  --index               Generate an index file for all produced fixtures.
+                        Specify a build name for the fixtures.ini file, e.g., 'stable'.
+  --skip-index SKIP_INDEX
+                        Skip generating an index file for all produced fixtures.
+  --block-gas-limit BLOCK_GAS_LIMIT
+                        Default gas limit used ceiling used for blocks and tests that attempt to consume
+                        an entire block's gas. (Default: 72000000)
 
 Arguments defining debug behavior:
-  --evm-dump-dir EVM_DUMP_DIR, --t8n-dump-dir EVM_DUMP_DIR
-                        Path to dump the transition tool debug output.  (Default: <repo>/logs/evm)
+  --evm-dump-dir, --t8n-dump-dir EVM_DUMP_DIR
+                        Path to dump the transition tool debug output. (Default:
+                        <repo>/logs/evm)
+  --skip-evm-dump, --skip-t8n-dump
+                        Skip dumping the the transition tool debug output.
 
 Specify the fork range to generate fixtures for:
   --forks               Display forks supported by the test framework and exit.
   --fork FORK           Only fill tests for the specified fork.
   --from FROM           Fill tests from and including the specified fork.
   --until UNTIL         Fill tests until and including the specified fork.
-
-Exit: After displaying help.
 ```

src/ethereum_test_specs/blockchain.py

@@ -1,5 +1,6 @@
 """Ethereum blockchain test spec definition and filler."""
 
+import warnings
 from pprint import pprint
 from typing import Any, Callable, ClassVar, Dict, Generator, List, Optional, Sequence, Tuple, Type
 
@@ -409,7 +410,15 @@ def generate_block_data(
         env = block.set_environment(previous_env)
         env = env.set_fork_requirements(fork)
 
-        txs = [tx.with_signature_and_sender() for tx in block.txs]
+        txs: List[Transaction] = []
+        for tx in block.txs:
+            if not self.is_slow_test() and tx.gas_limit >= Environment().gas_limit:
+                warnings.warn(
+                    f"{self.node_id()} uses a high Transaction gas_limit: {tx.gas_limit}",
+                    stacklevel=2,
+                )
+
+            txs.append(tx.with_signature_and_sender())
 
         if failing_tx_count := len([tx for tx in txs if tx.error]) > 0:
             if failing_tx_count > 1:

src/ethereum_test_specs/state.py

@@ -1,5 +1,6 @@
 """Ethereum state test spec definition and filler."""
 
+import warnings
 from pprint import pprint
 from typing import Any, Callable, ClassVar, Dict, Generator, List, Optional, Sequence, Type
 
@@ -169,6 +170,11 @@ def make_state_test_fixture(
 
         env = self.env.set_fork_requirements(fork)
         tx = self.tx.with_signature_and_sender(keep_secret_key=True)
+        if not self.is_slow_test() and tx.gas_limit >= Environment().gas_limit:
+            warnings.warn(
+                f"{self.node_id()} uses a high Transaction gas_limit: {tx.gas_limit}",
+                stacklevel=2,
+            )
         pre_alloc = Alloc.merge(
             Alloc.model_validate(fork.pre_allocation()),
             self.pre,

src/ethereum_test_specs/tests/test_fixtures.py

@@ -68,7 +68,7 @@ def test_check_helper_fixtures():
     ],
 )
 def test_make_genesis(fork: Fork, fixture_hash: bytes, default_t8n: TransitionTool):  # noqa: D103
-    env = Environment()
+    env = Environment(gas_limit=100_000_000_000_000_000)
 
     pre = Alloc(
         {
@@ -479,6 +479,7 @@ def post(self):  # noqa: D102
     @pytest.fixture
     def genesis_environment(self):  # noqa: D102
         return Environment(
+            gas_limit=100_000_000_000_000_000,
             base_fee_per_gas=1000,
             fee_recipient="0xba5e000000000000000000000000000000000000",
         )
@@ -869,6 +870,7 @@ def test_fill_blockchain_invalid_txs(
 
     # We start genesis with a baseFee of 1000
     genesis_environment = Environment(
+        gas_limit=100_000_000_000_000_000,
         base_fee_per_gas=1000,
         fee_recipient="0xba5e000000000000000000000000000000000000",
     )

src/ethereum_test_types/__init__.py

@@ -17,6 +17,7 @@
     ConsolidationRequest,
     DepositRequest,
     Environment,
+    EnvironmentDefaults,
     NetworkWrappedTransaction,
     Removable,
     Requests,
@@ -38,6 +39,7 @@
     "DepositRequest",
     "EmptyTrieRoot",
     "Environment",
+    "EnvironmentDefaults",
     "EOA",
     "Hash",
     "HeaderNonce",

src/ethereum_test_types/tests/test_types.py

@@ -4,7 +4,14 @@
 
 import pytest
 
-from ethereum_test_base_types import AccessList, Address, Bytes, TestPrivateKey, to_json
+from ethereum_test_base_types import (
+    AccessList,
+    Address,
+    Bytes,
+    TestPrivateKey,
+    ZeroPaddedHexNumber,
+    to_json,
+)
 from ethereum_test_base_types.pydantic import CopyValidateModel
 
 from ..types import (
@@ -451,7 +458,7 @@ def test_account_merge(
             Environment(),
             {
                 "currentCoinbase": "0x2adc25665018aa1fe0e6bc666dac8fc2697ff9ba",
-                "currentGasLimit": "0x016345785d8a0000",
+                "currentGasLimit": str(ZeroPaddedHexNumber(Environment().gas_limit)),
                 "currentNumber": "0x01",
                 "currentTimestamp": "0x03e8",
                 "blockHashes": {},
@@ -484,7 +491,7 @@ def test_account_merge(
             ),
             {
                 "currentCoinbase": "0x0000000000000000000000000000000000001234",
-                "currentGasLimit": "0x016345785d8a0000",
+                "currentGasLimit": str(ZeroPaddedHexNumber(Environment().gas_limit)),
                 "currentNumber": "0x01",
                 "currentTimestamp": "0x03e8",
                 "currentDifficulty": "0x05",

src/ethereum_test_types/types.py

@@ -344,6 +344,18 @@ class Withdrawal(WithdrawalGeneric[HexNumber]):
 
 
 DEFAULT_BASE_FEE = 7
+CURRENT_MAINNET_BLOCK_GAS_LIMIT = 36_000_000
+DEFAULT_BLOCK_GAS_LIMIT = CURRENT_MAINNET_BLOCK_GAS_LIMIT * 2
+
+
+@dataclass
+class EnvironmentDefaults:
+    """Default environment values."""
+
+    # By default, the constant `DEFAULT_BLOCK_GAS_LIMIT` is used.
+    # Other libraries (pytest plugins) may override this value by modifying the
+    # `EnvironmentDefaults.gas_limit` class attribute.
+    gas_limit: int = DEFAULT_BLOCK_GAS_LIMIT
 
 
 class EnvironmentGeneric(CamelModel, Generic[NumberBoundTypeVar]):
@@ -353,7 +365,9 @@ class EnvironmentGeneric(CamelModel, Generic[NumberBoundTypeVar]):
         Address("0x2adc25665018aa1fe0e6bc666dac8fc2697ff9ba"),
         alias="currentCoinbase",
     )
-    gas_limit: NumberBoundTypeVar = Field(100_000_000_000_000_000, alias="currentGasLimit")  # type: ignore
+    gas_limit: NumberBoundTypeVar = Field(
+        default_factory=lambda: EnvironmentDefaults.gas_limit, alias="currentGasLimit"
+    )  # type: ignore
     number: NumberBoundTypeVar = Field(1, alias="currentNumber")  # type: ignore
     timestamp: NumberBoundTypeVar = Field(1_000, alias="currentTimestamp")  # type: ignore
     prev_randao: NumberBoundTypeVar | None = Field(None, alias="currentRandom")

src/pytest_plugins/execute/execute.py

@@ -11,7 +11,7 @@
 from ethereum_test_forks import Fork
 from ethereum_test_rpc import EthRPC
 from ethereum_test_tools import SPEC_TYPES, BaseTest
-from ethereum_test_types import TransactionDefaults
+from ethereum_test_types import EnvironmentDefaults, TransactionDefaults
 from pytest_plugins.spec_version_checker.spec_version_checker import EIPSpecTestItem
 
 from ..shared.helpers import get_spec_format_for_item, labeled_format_parameter_set
@@ -56,6 +56,18 @@ def pytest_addoption(parser):
             "unless overridden by the test."
         ),
     )
+    execute_group.addoption(
+        "--transaction-gas-limit",
+        action="store",
+        dest="transaction_gas_limit",
+        default=EnvironmentDefaults.gas_limit // 4,
+        type=int,
+        help=(
+            "Maximum gas used to execute a single transaction. "
+            "Will be used as ceiling for tests that attempt to consume the entire block gas limit."
+            f"(Default: {EnvironmentDefaults.gas_limit // 4})"
+        ),
+    )
 
     report_group = parser.getgroup("tests", "Arguments defining html report behavior")
     report_group.addoption(
@@ -86,6 +98,9 @@ def pytest_configure(config):
         called before the pytest-html plugin's pytest_configure to ensure that
         it uses the modified `htmlpath` option.
     """
+    # Modify the block gas limit if specified.
+    if config.getoption("transaction_gas_limit"):
+        EnvironmentDefaults.gas_limit = config.getoption("transaction_gas_limit")
     if config.option.collectonly:
         return
     if config.getoption("disable_html") and config.getoption("htmlpath") is None:

src/pytest_plugins/filler/filler.py

@@ -31,6 +31,7 @@
     generate_github_url,
     get_current_commit_hash_or_tag,
 )
+from ethereum_test_types import EnvironmentDefaults
 
 from ..shared.helpers import get_spec_format_for_item, labeled_format_parameter_set
 
@@ -172,6 +173,17 @@ def pytest_addoption(parser: pytest.Parser):
         default=True,
         help="Skip generating an index file for all produced fixtures.",
     )
+    test_group.addoption(
+        "--block-gas-limit",
+        action="store",
+        dest="block_gas_limit",
+        default=EnvironmentDefaults.gas_limit,
+        type=int,
+        help=(
+            "Default gas limit used ceiling used for blocks and tests that attempt to "
+            f"consume an entire block's gas. (Default: {EnvironmentDefaults.gas_limit})"
+        ),
+    )
 
     debug_group = parser.getgroup("debug", "Arguments defining debug behavior")
     debug_group.addoption(
@@ -211,6 +223,9 @@ def pytest_configure(config):
         called before the pytest-html plugin's pytest_configure to ensure that
         it uses the modified `htmlpath` option.
     """
+    # Modify the block gas limit if specified.
+    if config.getoption("block_gas_limit"):
+        EnvironmentDefaults.gas_limit = config.getoption("block_gas_limit")
     if config.option.collectonly:
         return
     if not config.getoption("disable_html") and config.getoption("htmlpath") is None: