Skip to content

Commit 3ecad5c

Browse files
danceratopzdanceratopz
committed
docs: some ideas on moving ./tests to ethereum/execution-specs
1 parent bceb08f commit 3ecad5c

File tree

1 file changed

+117
-0
lines changed

1 file changed

+117
-0
lines changed

‎docs/internal/moving_tests_to_execution_specs.md

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
# Moving `./tests/` to `execution-specs`
2+
3+
## Requirements
4+
5+
1. Test implementers can continue working on tests from Day 1.
6+
i. TBD: Test implementers can continue using familiar tooling: `uv`, `ruff`, `tox`.
7+
ii. Familiar CI checks enabled in Github Actions.
8+
2. The execution-spec-tests git history of `./tests` is maintained in ethereum-specs.
9+
3. Consistent pytest versions across repos.
10+
4. Consistent minimum Python versions across repos.
11+
5. The "[Test Case Reference](https://eest.ethereum.org/main/tests/)" documentation is published on ethereum-specs.
12+
13+
## Main Steps
14+
15+
1. EEST: Make EEST an installable package, tracking issue: [ethereum/execution-spec-tests#1565](https://github.com/ethereum/execution-spec-tests/issues/1565).
16+
2. EEST: Remove last dependencies on execution-specs to remove any circular dependencies. These are use of `FrontierAccount` and `FrontierAddress` for state root calculation in [ethereum_test_types](https://github.com/ethereum/execution-spec-tests/blob/d9f5eabebb35b4f6a44e10f9163aa27e31125c52/src/ethereum_test_types/types.py#L222-L245).
17+
3. TBD, EELS: Update execution-specs to
18+
4. New repo: Add shared Github Actions for building clients (e.g., evmone-t8n required for coverage flow).
19+
5. EELS & EEST: Enable execution-spec-tests CI flows in execution-specs.
20+
6. EELS: Add Github Pages at eels.ethereum.org.
21+
7. EELS: Add mkdocs doc flow to publish Test Case Reference.
22+
23+
## Questions
24+
25+
1. Do we need a new top-level `steel` Github organisation? This could host components that don't belong in either repo, e.g.,
26+
- [marioevz/hive.py](https://github.com/marioevz/hive.py)
27+
- [petertdavies/ethereum-spec-evm-resolver](https://github.com/petertdavies/ethereum-spec-evm-resolver)
28+
- Other components, such as pytest plugins.
29+
3. TBD: Should execution-specs adopt conventional commits similar to execution-spec-tests?
30+
2. Low prio: EIP versioning: Should the implementation also define MD5 hash digests of the ethereum/EIPs markdown?
31+
32+
## `ethereum/execution-spec-tests` Component Analysis
33+
34+
### 1. GitHub Workflows and Actions
35+
36+
| Component Name | Description | Repository Path | Future Location |
37+
| ----------------------- | ----------------------------------------------------------------------------------- | ----------------------------------------- | --------------- |
38+
| check_eip_versions.yaml | Checks if EIP version references in test code are outdated, creates issues if found | .github/workflows/check_eip_versions.yaml | execution-specs |
39+
| check_links.yml | Validates links in documentation and source code using lychee | .github/workflows/check_links.yml | EEST |
40+
| coverage.yaml | Generates evmone coverage reports for test changes in PRs | .github/workflows/coverage.yaml | execution-specs |
41+
| docs_deploy.yaml | Deploys documentation to GitHub Pages using mkdocs | .github/workflows/docs_deploy.yaml | execution-specs |
42+
| fixtures_feature.yaml | Builds fixtures for specific features based on tags | .github/workflows/fixtures_feature.yaml | execution-specs |
43+
| fixtures.yaml | Builds fixtures for standard forks on version tags | .github/workflows/fixtures.yaml | execution-specs |
44+
| tox_verify.yaml | Main CI workflow for code quality checks (lint, type check, tests) | .github/workflows/tox_verify.yaml | Split/Both; see `tox` table below |
45+
| build-evm-base | Base action for building EVM clients from configuration | .github/actions/build-evm-base/ | new repo |
46+
| build-fixtures | Builds and packages test fixtures for release | .github/actions/build-fixtures/ | execution-specs |
47+
| build-evm-client/evmone | Builds evmone EVM implementation | .github/actions/build-evm-client/evmone/ | new repo |
48+
| build-evm-client/geth | Builds Go-Ethereum EVM implementation | .github/actions/build-evm-client/geth/ | new repo |
49+
| build-evm-client/besu | Builds Hyperledger Besu EVM implementation | .github/actions/build-evm-client/besu/ | new repo |
50+
| build-evm-client/ethjs | Builds EthereumJS EVM implementation | .github/actions/build-evm-client/ethjs/ | new repo |
51+
52+
### 2. Command-Line Interfaces (CLIs)
53+
54+
| Component Name | Description | Repository Path | Future Location |
55+
| ------------------ | --------------------------------------------- | --------------------------------------------- | --------------- |
56+
| fill | Fills test cases using pytest wrapper | src/cli/pytest_commands/fill.py | EEST |
57+
| execute | Executes test cases against clients | src/cli/pytest_commands/execute.py | EEST |
58+
| consume | Validates test fixtures against clients | src/cli/pytest_commands/consume.py | EEST |
59+
| checkfixtures | Checks test fixtures for validity | src/cli/check_fixtures.py | delete |
60+
| check_eip_versions | Verifies EIP references are up-to-date | src/cli/pytest_commands/check_eip_versions.py | EEST |
61+
| genindex | Generates index for test fixtures | src/cli/gen_index.py | EEST; used in execution-specs |
62+
| gentest | Generates test templates from specifications | src/cli/gentest/ | EEST; used in execution-specs |
63+
| eofwrap | Wraps bytecode in EOF format | src/cli/eofwrap.py | ? |
64+
| order_fixtures | Orders fixture files appropriately | src/cli/order_fixtures.py | ? |
65+
| evm_bytes | Handles EVM bytecode operations | src/cli/evm_bytes.py | EEST |
66+
| hasher | Creates hashes for test fixtures verification | src/cli/hasher.py | EEST |
67+
| eest info | Print repo info and tool versions | src/cli/eest/ | both; add `eels` command? |
68+
| eest clean | Remove temporary files/artifacts | src/cli/eest/ | both |
69+
| eest make | Create a new test template | src/cli/eest/ | Move `make` to `es` command? |
70+
| fillerconvert | Converts legacy test fillers to new format | src/cli/fillerconvert/ | ? |
71+
72+
### 3. Top-level Library Functions
73+
74+
| Component Name | Description | Repository Path | Future Location |
75+
| ------------------------ | --------------------------------------------------------- | ----------------------------- | --------------- |
76+
| ethereum_test_base_types | Base types for Ethereum testing (addresses, hashes, etc.) | src/ethereum_test_base_types/ | |
77+
| ethereum_test_exceptions | Exception mapping and handling for test scenarios | src/ethereum_test_exceptions/ | ? |
78+
| ethereum_test_execution | Execution context and transaction handling | src/ethereum_test_execution/ | |
79+
| ethereum_test_fixtures | Test fixture formats and consumers | src/ethereum_test_fixtures/ | |
80+
| ethereum_test_forks | Fork definitions and capabilities | src/ethereum_test_forks/ | |
81+
| ethereum_test_rpc | RPC client interfaces for testing | src/ethereum_test_rpc/ | |
82+
| ethereum_test_specs | Test specification classes (state, blockchain) | src/ethereum_test_specs/ | |
83+
| ethereum_test_tools | Utility functions for test creation | src/ethereum_test_tools/ | |
84+
| ethereum_test_types | Common types used across test scenarios | src/ethereum_test_types/ | |
85+
| ethereum_test_vm | Virtual machine operations and opcodes | src/ethereum_test_vm/ | |
86+
| ethereum_clis | Client interfaces for various Ethereum implementations | src/ethereum_clis/ | EEST |
87+
88+
### 4. Pytest Plugins
89+
90+
| Component Name | Description | Repository Path | Future Location |
91+
| -------------------- | ------------------------------------------------------- | ---------------------------------------- | --------------- |
92+
| consume | Consumes and validates generated test fixtures | src/pytest_plugins/consume/ | |
93+
| execute | Executes test cases against Ethereum clients | src/pytest_plugins/execute/ | |
94+
| filler | Fills test templates to generate fixtures | src/pytest_plugins/filler/ | |
95+
| gen_test_docs | Generates reference test case documentation | src/pytest_plugins/filler/ | |
96+
| forks | Handles fork-specific test parametrization | src/pytest_plugins/forks/ | |
97+
| help | Provides help command integration for pytest | src/pytest_plugins/help/ | |
98+
| logging | Enhanced logging with custom levels and features | src/pytest_plugins/logging/ | |
99+
| spec_version_checker | Verifies EIP specification versions | src/pytest_plugins/spec_version_checker/ | |
100+
| solc | Manages Solidity compiler integration | src/pytest_plugins/solc/ | |
101+
| pytest_hive | Integration with Hive testing framework | src/pytest_plugins/pytest_hive/ | |
102+
| concurrency | Manages concurrent test execution | src/pytest_plugins/concurrency.py | |
103+
| eels_resolver | Resolves EELS (Ethereum Execution Layer Specifications) | src/pytest_plugins/eels_resolver.py | |
104+
105+
### 5. Tox Environments
106+
107+
| Component Name | Description | Repository Path | Future Location |
108+
| -------------------- | --------------------------------------------------------------------------------- | --------------- | --------------- |
109+
| lint | Runs ruff for code formatting and linting checks | tox.ini | Both |
110+
| typecheck | Runs mypy for type checking Python code | tox.ini | Both |
111+
| markdownlint | Lints markdown files for formatting issues | tox.ini | Both |
112+
| spellcheck | Checks spelling in documentation using pyspelling | tox.ini | Both |
113+
| pytest | Runs framework unit tests and library tests | tox.ini | EEST |
114+
| tests-deployed | Fills test cases for deployed mainnet forks (excludes slow/zkevm) | tox.ini | execution-specs |
115+
| tests-deployed-zkevm | Fills zkEVM test cases using evmone-t8n | tox.ini | execution-specs |
116+
| mkdocs | Builds documentation in strict mode using mkdocs | tox.ini | execution-specs |
117+
| tests-develop | Fills test cases for development forks | tox.ini | execution-specs |

0 commit comments

Comments
 (0)