Skip to content

Commit e673aba

Browse files
danceratopzdanceratopz
committed
docs: some ideas on moving ./tests to ethereum/execution-specs
1 parent bceb08f commit e673aba

File tree

2 files changed

+138
-0
lines changed

2 files changed

+138
-0
lines changed

‎docs/internal/moving_tests_to_execution_specs.md

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
# Moving `./tests/` to `execution-specs`
2+
3+
## Requirements
4+
5+
1. Test implementers can continue working on tests from Day 1.
6+
i. Familiar CI checks enabled in Github Actions.
7+
ii. TBD: Test implementers can continue using familiar tooling: `uv`, `ruff`, `tox`.
8+
2. The execution-spec-tests git history of `./tests` is maintained in ethereum-specs.
9+
3. Consistent pytest versions across repos.
10+
4. Consistent minimum Python versions across repos.
11+
5. The "[Test Case Reference](https://eest.ethereum.org/main/tests/)" documentation is published on ethereum-specs.
12+
13+
## Main Steps
14+
15+
### Phase I
16+
17+
For the initial move.
18+
19+
1. EEST: Make EEST an installable package, tracking issue: [ethereum/execution-spec-tests#1565](https://github.com/ethereum/execution-spec-tests/issues/1565).
20+
2. EEST: Remove last dependencies on execution-specs to remove any circular dependencies:
21+
- Use of `FrontierAccount` and `FrontierAddress` for state root calculation in [ethereum_test_types](https://github.com/ethereum/execution-spec-tests/blob/d9f5eabebb35b4f6a44e10f9163aa27e31125c52/src/ethereum_test_types/types.py#L222-L245).
22+
3. TBD, EELS: Update execution-specs to same Python tooling experience (`uv`, `ruff`, ...) as EEST?
23+
4. New repo: Add shared Github Actions for building clients (e.g., evmone-t8n required for coverage flow).
24+
5. EELS & EEST: Enable execution-spec-tests CI flows in execution-specs.
25+
6. EELS: Add Github Pages at eels.ethereum.org.
26+
7. EELS: Add mkdocs doc flow to publish a basic doc structure and the Test Case Reference.
27+
28+
### Phase II
29+
30+
Can be left as follow-up work.
31+
32+
1. TBD: EEST: Move some `src/ethereum_test*` libraries to EELS or a shared repo?
33+
2. TBD: EEST: Publish pytest plugins individually in separate repos?
34+
35+
## Other Questions (than the TBDs above)
36+
37+
1. Do we need a new top-level `steel` Github organization? This could host components that don't belong in either repo, e.g.,
38+
- [marioevz/hive.py](https://github.com/marioevz/hive.py)
39+
- [petertdavies/ethereum-spec-evm-resolver](https://github.com/petertdavies/ethereum-spec-evm-resolver)
40+
- Other components, such as pytest plugins.
41+
2. TBD: Should execution-specs adopt conventional commits similar to execution-spec-tests?
42+
3. Low priority: EIP versioning: Should the implementation also define MD5 hash digests of the ethereum/EIPs markdown?
43+
44+
## `ethereum/execution-spec-tests` Component Analysis
45+
46+
### 1. GitHub Workflows and Actions
47+
48+
| Component Name | Description | Repository Path | Future Location |
49+
| ----------------------- | ----------------------------------------------------------------------------------- | ----------------------------------------- | --------------------------------- |
50+
| check_eip_versions.yaml | Checks if EIP version references in test code are outdated, creates issues if found | .github/workflows/check_eip_versions.yaml | execution-specs |
51+
| check_links.yml | Validates links in documentation and source code using lychee | .github/workflows/check_links.yml | EEST |
52+
| coverage.yaml | Generates evmone coverage reports for test changes in PRs | .github/workflows/coverage.yaml | execution-specs |
53+
| docs_deploy.yaml | Deploys documentation to GitHub Pages using mkdocs | .github/workflows/docs_deploy.yaml | execution-specs |
54+
| fixtures_feature.yaml | Builds fixtures for specific features based on tags | .github/workflows/fixtures_feature.yaml | execution-specs |
55+
| fixtures.yaml | Builds fixtures for standard forks on version tags | .github/workflows/fixtures.yaml | execution-specs |
56+
| tox_verify.yaml | Main CI workflow for code quality checks (lint, type check, tests) | .github/workflows/tox_verify.yaml | Split/Both; see `tox` table below |
57+
| build-evm-base | Base action for building EVM clients from configuration | .github/actions/build-evm-base/ | new repo |
58+
| build-fixtures | Builds and packages test fixtures for release | .github/actions/build-fixtures/ | execution-specs |
59+
| build-evm-client/evmone | Builds evmone EVM implementation | .github/actions/build-evm-client/evmone/ | new repo |
60+
| build-evm-client/geth | Builds Go-Ethereum EVM implementation | .github/actions/build-evm-client/geth/ | new repo |
61+
| build-evm-client/besu | Builds Hyperledger Besu EVM implementation | .github/actions/build-evm-client/besu/ | new repo |
62+
| build-evm-client/ethjs | Builds EthereumJS EVM implementation | .github/actions/build-evm-client/ethjs/ | new repo |
63+
64+
### 2. Tox Environments
65+
66+
| Component Name | Description | Repository Path | Future Location |
67+
| -------------------- | ----------------------------------------------------------------- | --------------- | --------------- |
68+
| lint | Runs ruff for code formatting and linting checks | tox.ini | Both |
69+
| typecheck | Runs mypy for type checking Python code | tox.ini | Both |
70+
| markdownlint | Lints markdown files for formatting issues | tox.ini | Both |
71+
| spellcheck | Checks spelling in documentation using pyspelling | tox.ini | Both |
72+
| pytest | Runs framework unit tests and library tests | tox.ini | EEST |
73+
| tests-deployed | Fills test cases for deployed mainnet forks (excludes slow/zkevm) | tox.ini | execution-specs |
74+
| tests-deployed-zkevm | Fills zkEVM test cases using evmone-t8n | tox.ini | execution-specs |
75+
| mkdocs | Builds documentation in strict mode using mkdocs | tox.ini | execution-specs |
76+
| tests-develop | Fills test cases for development forks | tox.ini | execution-specs |
77+
78+
### 3. Command-Line Interfaces (CLIs)
79+
80+
| Component Name | Description | Repository Path | Future Location |
81+
| ------------------ | --------------------------------------------- | --------------------------------------------- | ----------------------------- |
82+
| fill | Fills test cases using pytest wrapper | src/cli/pytest_commands/fill.py | EEST |
83+
| execute | Executes test cases against clients | src/cli/pytest_commands/execute.py | EEST |
84+
| consume | Validates test fixtures against clients | src/cli/pytest_commands/consume.py | EEST |
85+
| checkfixtures | Checks test fixtures for validity | src/cli/check_fixtures.py | delete |
86+
| check_eip_versions | Verifies EIP references are up-to-date | src/cli/pytest_commands/check_eip_versions.py | EEST |
87+
| genindex | Generates index for test fixtures | src/cli/gen_index.py | EEST; used in execution-specs |
88+
| gentest | Generates test templates from specifications | src/cli/gentest/ | EEST; used in execution-specs |
89+
| eofwrap | Wraps bytecode in EOF format | src/cli/eofwrap.py | ? |
90+
| order_fixtures | Orders fixture files appropriately | src/cli/order_fixtures.py | ? |
91+
| evm_bytes | Handles EVM bytecode operations | src/cli/evm_bytes.py | EEST |
92+
| hasher | Creates hashes for test fixtures verification | src/cli/hasher.py | EEST |
93+
| eest info | Print repo info and tool versions | src/cli/eest/ | both; add `eels` command? |
94+
| eest clean | Remove temporary files/artifacts | src/cli/eest/ | both |
95+
| eest make | Create a new test template | src/cli/eest/ | Move `make` to `es` command? |
96+
| fillerconvert | Converts legacy test fillers to new format | src/cli/fillerconvert/ | ? |
97+
98+
### 4. Top-level Library Functions
99+
100+
| Component Name | Description | Repository Path | Future Location |
101+
| ------------------------ | --------------------------------------------------------- | ----------------------------- | --------------- |
102+
| ethereum_test_base_types | Base types for Ethereum testing (addresses, hashes, etc.) | src/ethereum_test_base_types/ | |
103+
| ethereum_test_exceptions | Exception mapping and handling for test scenarios | src/ethereum_test_exceptions/ | ? |
104+
| ethereum_test_execution | Execution context and transaction handling | src/ethereum_test_execution/ | |
105+
| ethereum_test_fixtures | Test fixture formats and consumers | src/ethereum_test_fixtures/ | |
106+
| ethereum_test_forks | Fork definitions and capabilities | src/ethereum_test_forks/ | |
107+
| ethereum_test_rpc | RPC client interfaces for testing | src/ethereum_test_rpc/ | |
108+
| ethereum_test_specs | Test specification classes (state, blockchain) | src/ethereum_test_specs/ | |
109+
| ethereum_test_tools | Utility functions for test creation | src/ethereum_test_tools/ | |
110+
| ethereum_test_types | Common types used across test scenarios | src/ethereum_test_types/ | |
111+
| ethereum_test_vm | Virtual machine operations and opcodes | src/ethereum_test_vm/ | |
112+
| ethereum_clis | Client interfaces for various Ethereum implementations | src/ethereum_clis/ | EEST |
113+
114+
### 5. Pytest Plugins
115+
116+
| Component Name | Description | Repository Path | Future Location |
117+
| -------------------- | ------------------------------------------------------- | ---------------------------------------- | --------------- |
118+
| consume | Consumes and validates generated test fixtures | src/pytest_plugins/consume/ | EEST |
119+
| execute | Executes test cases against Ethereum clients | src/pytest_plugins/execute/ | EEST |
120+
| filler | Fills test templates to generate fixtures | src/pytest_plugins/filler/ | EEST |
121+
| gen_test_docs | Generates reference test case documentation | src/pytest_plugins/filler/ | EEST |
122+
| forks | Handles fork-specific test parametrization | src/pytest_plugins/forks/ | EEST |
123+
| help | Provides help command integration for pytest | src/pytest_plugins/help/ | EEST |
124+
| logging | Enhanced logging with custom levels and features | src/pytest_plugins/logging/ | EEST |
125+
| spec_version_checker | Verifies EIP specification versions | src/pytest_plugins/spec_version_checker/ | EEST |
126+
| solc | Manages Solidity compiler integration | src/pytest_plugins/solc/ | EEST |
127+
| pytest_hive | Integration with Hive testing framework | src/pytest_plugins/pytest_hive/ | EEST |
128+
| concurrency | Manages concurrent test execution | src/pytest_plugins/concurrency.py | EEST |
129+
| eels_resolver | Resolves EELS (Ethereum Execution Layer Specifications) | src/pytest_plugins/eels_resolver.py | EEST |

‎whitelist.txt

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1006,3 +1006,12 @@ Typecheck
10061006
autoformat
10071007
Typechecking
10081008
zkEVMs
1009+
implementers
1010+
TBDs
1011+
ethjs
1012+
lychee
1013+
typecheck
1014+
zkevm
1015+
checkfixtures
1016+
fillerconvert
1017+
genindex

0 commit comments

Comments
 (0)