You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The full execution-spec-tests documentation can be found [here](https://eest.ethereum.org/main/).
10
+
9
11
[ethereum/execution-spec-tests](https://github.com/ethereum/execution-spec-tests) is both a collection of test cases and a framework implemented in Python to generate tests for Ethereum execution clients.
10
12
11
13
The framework collects and executes the test cases in order to generate _test fixtures_ (JSON) which can be consumed by any execution client to verify their implementation of [ethereum/execution-specs](https://github.com/ethereum/execution-specs). The fixtures, which define state transition and block tests, are generated by the framework using one of the `t8n` command-line tools that are provided by most execution clients, see below for an overview of the supported `t8n` tools.
Copy file name to clipboardExpand all lines: docs/CHANGELOG.md
+1Lines changed: 1 addition & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -57,6 +57,7 @@ Users can select any of the artifacts depending on their testing needs for their
57
57
58
58
### 📋 Misc
59
59
60
+
- 🔀 Misc. doc updates, including a navigation footer ([#1846](https://github.com/ethereum/execution-spec-tests/pull/1846)).
60
61
- 🔀 Remove Python 3.10 support ([#1808](https://github.com/ethereum/execution-spec-tests/pull/1808)).
61
62
- 🔀 Modernize codebase with Python 3.11 language features ([#1812](https://github.com/ethereum/execution-spec-tests/pull/1812)).
62
63
- ✨ Add changelog formatting validation to CI to ensure consistent punctuation in bullet points [#1691](https://github.com/ethereum/execution-spec-tests/pull/1691).
Copy file name to clipboardExpand all lines: docs/filling_tests/filling_tests_command_line.md
+18-18Lines changed: 18 additions & 18 deletions
Original file line number
Diff line number
Diff line change
@@ -6,7 +6,7 @@ The execution-spec-tests test framework uses the [pytest framework](https://docs
6
6
The command-line options specific to filling tests can be listed via:
7
7
8
8
```console
9
-
fill --help
9
+
uv run fill --help
10
10
```
11
11
12
12
See [Custom `fill` Command-Line Options](#custom-fill-command-line-options) for all options.
@@ -16,35 +16,35 @@ The execution-spec-tests test framework uses the [pytest framework](https://docs
16
16
The test cases implemented in the `./tests` sub-directory can be listed in the console using:
17
17
18
18
```console
19
-
fill --collect-only
19
+
uv run fill --collect-only
20
20
```
21
21
22
22
and can be filtered (by test path, function and parameter substring):
23
23
24
24
```console
25
-
fill --collect-only -k warm_coinbase
25
+
uv run fill --collect-only -k warm_coinbase
26
26
```
27
27
28
28
Docstrings are additionally displayed when ran verbosely:
29
29
30
30
```console
31
-
fill --collect-only -k warm_coinbase -vv
31
+
uv run fill --collect-only -k warm_coinbase -vv
32
32
```
33
33
34
34
## Execution
35
35
36
36
By default, test cases are filled for all forks already deployed to mainnet, but not for forks still under active development, i.e., as of time of writing, Q2 2023:
37
37
38
38
```console
39
-
fill
39
+
uv run fill
40
40
```
41
41
42
42
will generate fixtures for test cases from Frontier to Shanghai.
43
43
44
44
To generate all the test fixtures defined in the `./tests/shanghai` sub-directory and write them to the `./fixtures-shanghai` directory, run `fill` in the top-level directory as:
45
45
46
46
```console
47
-
fill ./tests/shanghai --output="fixtures-shanghai"
47
+
uv run fill ./tests/shanghai --output="fixtures-shanghai"
48
48
```
49
49
50
50
!!! note "Test case verification"
@@ -53,25 +53,25 @@ fill ./tests/shanghai --output="fixtures-shanghai"
53
53
To generate all the test fixtures in the `tests/shanghai/eip3651_warm_coinbase/test_warm_coinbase.py` module, for example, run:
54
54
55
55
```console
56
-
fill tests/shanghai/eip3651_warm_coinbase/test_warm_coinbase.py
56
+
uv run fill tests/shanghai/eip3651_warm_coinbase/test_warm_coinbase.py
57
57
```
58
58
59
59
To generate specific test fixtures from a specific test function or even test function and parameter set, obtain the corresponding test ID using:
60
60
61
61
```console
62
-
fill --collect-only -q -k test_warm_coinbase
62
+
uv run fill --collect-only -q -k test_warm_coinbase
63
63
```
64
64
65
65
This filters the tests by `test_warm_coinbase`. Then find the relevant test ID in the console output and provide it to fill, for example, for a test function:
66
66
67
67
```console
68
-
fill tests/shanghai/eip3651_warm_coinbase/test_warm_coinbase.py::test_warm_coinbase_gas_usage
68
+
uv run fill tests/shanghai/eip3651_warm_coinbase/test_warm_coinbase.py::test_warm_coinbase_gas_usage
69
69
```
70
70
71
71
or, for a test function and specific parameter combination:
72
72
73
73
```console
74
-
fill tests/shanghai/eip3651_warm_coinbase/test_warm_coinbase.py::test_warm_coinbase_gas_usage[fork_Paris-DELEGATECALL]
74
+
uv run fill tests/shanghai/eip3651_warm_coinbase/test_warm_coinbase.py::test_warm_coinbase_gas_usage[fork_Paris-DELEGATECALL]
75
75
```
76
76
77
77
## Execution for Development Forks
@@ -80,10 +80,10 @@ fill tests/shanghai/eip3651_warm_coinbase/test_warm_coinbase.py::test_warm_coinb
80
80
By default, test cases are not filled for upcoming Ethereum forks so that they can be readily filled using the `evm` tool from the latest `geth` release.
81
81
82
82
In order to fill test cases for an upcoming fork, ensure that the `evm` tool used supports that fork and features under test and use the `--until` or `--fork` flag.
83
-
83
+
84
84
For example, as of Q2 2023, the current fork under active development is `Cancun`:
85
85
```console
86
-
fill --until Cancun
86
+
uv run fill --until Cancun
87
87
```
88
88
89
89
See: [Filling Tests for Features under Development](./filling_tests_dev_fork.md).
@@ -95,10 +95,10 @@ The `--evm-dump-dir` flag can be used to dump the inputs and outputs of every ca
95
95
## Other Useful Pytest Command-Line Options
96
96
97
97
```console
98
-
fill -vv # More verbose output
99
-
fill -x # Exit instantly on first error or failed test case
100
-
fill --pdb -nauto # Drop into the debugger upon error in a test case
101
-
fill -s # Print stdout from tests to the console during execution
98
+
uv run fill -vv # More verbose output
99
+
uv run fill -x # Exit instantly on first error or failed test case
100
+
uv run fill --pdb -nauto # Drop into the debugger upon error in a test case
101
+
uv run fill -s # Print stdout from tests to the console during execution
102
102
```
103
103
104
104
## Custom `fill` Command-Line Options
@@ -108,7 +108,7 @@ To see all the options available to fill, including pytest and pytest plugin opt
108
108
To list the options that only specific to fill, use:
109
109
110
110
```console
111
-
fill --help
111
+
uv run fill --help
112
112
```
113
113
114
-
For a complete, up-to-date list of all command-line options, see the [Fill Command-Line Options](filling_tests_command_line_options.md) page, which is automatically generated from the current `fill --help` output.
114
+
For a complete, up-to-date list of all command-line options, see the [Fill Command-Line Options](filling_tests_command_line_options.md) page, which is automatically generated from the current `uv run fill --help` output.
Copy file name to clipboardExpand all lines: docs/running_tests/releases.md
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -26,7 +26,7 @@ Please see below for an explanation of the optional `<pre_release_name>` that is
26
26
27
27
### Standard Releases
28
28
29
-
Releases are published on the @ethereum/execution-spec-tests[releases](https://github.com/ethereum/execution-spec-tests/releases) page. Standard releases are tagged using the format `vX.Y.Z` (they don't have a don't `<pre_release_name>`).
29
+
Releases are published on the @ethereum/execution-spec-tests[releases](https://github.com/ethereum/execution-spec-tests/releases) page. Standard releases are tagged using the format `vX.Y.Z` (they don't have a `<pre_release_name>`).
30
30
31
31
For standard releases, two tarballs are available:
Copy file name to clipboardExpand all lines: docs/running_tests/running.md
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -110,7 +110,7 @@ See [Execute Command](./execute/index.md).
110
110
111
111
## Two Methods to Run EEST Simulators
112
112
113
-
Many of the methods use the Hive Testing Environment to interact clients and run tests against them. These methods are also called Hive simulators. While Hive is always necessary to run simulators, they can be called in two different ways. Both of these commands execute the same simulator code, but in different environments, we take the example of the `eest/consume-engine` simulator:
113
+
Many of the methods use the Hive Testing Environment to interact with clients and run tests against them. These methods are also called Hive simulators. While Hive is always necessary to run simulators, they can be called in two different ways. Both of these commands execute the same simulator code, but in different environments, we take the example of the `eest/consume-engine` simulator:
114
114
115
115
1.`./hive --sim=eest/consume-engine` is a standalone command that installs EEST and the `consume` command in a dockerized container managed by Hive. This is the standard method to execute EEST [fixture releases](./releases.md) against clients in CI environments and is the method to generate the results at [hive.ethpandaops.io](https://hive.ethpandaops.io). See [Hive](./hive/index.md) and its [Common Options](./hive/common_options.md) for help with this method.
116
116
2.`uv run consume engine` requires the user to clone and [install EEST](../getting_started/installation.md) and start a Hive server in [development mode](./hive/dev_mode.md). In this case, the simulator runs on the native system and communicate to the client via the Hive API. This is particularly useful during test development as fixtures on the local disk can be specified via `--input=fixtures/`. As the simulator runs natively, it is easy to drop into a debugger and inspect the simulator or client container state. See [Hive Developer Mode](./hive/dev_mode.md) for help with this method.
0 commit comments