feat(coverage): coverage script parsing files from ported_from markers (ethereum#1634)

* add missing ported_from markers

* adjust the coverage script to read from .py markers

* remove converted-ethereum-tests.txt

* fix empty ported list scenario

* register diffPlaces in stSelfBalance for coverage

* replace pyscript with fill plugin

* chore(fill): exit early if in a help/collectonly mode

* chore(fill): don't print ported_tests terminal summary with `-q`

* address comments

* docs: update changelog

---------

Co-authored-by: danceratopz <danceratopz@gmail.com>

Author: 2 people

.github/PULL_REQUEST_TEMPLATE.md

@@ -10,8 +10,8 @@
 - [ ] All: Considered squashing commits to improve commit history.
 - [ ] All: Added an entry to [CHANGELOG.md](/ethereum/execution-spec-tests/blob/main/docs/CHANGELOG.md).
 - [ ] All: Considered updating the online docs in the [./docs/](/ethereum/execution-spec-tests/blob/main/docs/) directory.
-- [ ] Tests: All converted JSON/YML tests from [ethereum/tests](/ethereum/tests) have been added to [converted-ethereum-tests.txt](/ethereum/execution-spec-tests/blob/main/converted-ethereum-tests.txt).
-- [ ] Tests: A PR with removal of converted JSON/YML tests from [ethereum/tests](/ethereum/tests) have been opened.
+- [ ] Tests: All converted JSON/YML tests from [ethereum/tests](/ethereum/tests)/[tests/static](/ethereum/execution-spec-tests/blob/main/tests/static) have been assigned @ported_from marker.
+- [ ] Tests: A PR with removal of converted JSON/YML blockchain tests from [ethereum/tests](/ethereum/tests) have been opened.
 - [ ] Tests: Included the type and version of evm t8n tool used to locally execute test cases:  e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
 - [ ] Tests: Ran `mkdocs serve` locally and verified the auto-generated docs for new tests in the [Test Case Reference](https://eest.ethereum.org/main/tests/) are correctly formatted.
 - [ ] Tests: For PRs implementing a missed test case, update the [post-mortem document](/ethereum/execution-spec-tests/blob/main/docs/writing_tests/post_mortems.md) to add an entry the list.

.github/workflows/coverage.yaml

@@ -36,8 +36,6 @@ jobs:
               - tests/**/test_*.py
               - '!tests/prague/**'
               - '!tests/unscheduled/**'
-            converted_tests:
-              - converted-ethereum-tests.txt
 
       - name: Exit workflow if there are no changed python files
         if: steps.changed-tests.outputs.tests_any_changed != 'true'
@@ -113,23 +111,22 @@ jobs:
             Cancun/BlockchainTests
 
       # This command diffs the file and filters in new lines
-      - name: Parse converted tests from converted-ethereum-tests.txt
+      - name: Parse ported_from markers from introduced .py tests
         if: steps.changed-tests.outputs.tests_any_changed == 'true'
+        env:
+          CHANGED_TEST_FILES: ${{ steps.changed-tests.outputs.tests_all_changed_files }}
         run: |
-          echo "New lines introduced in converted-ethereum-tests.txt:"
-          lines=$(git diff origin/${{ github.base_ref }} HEAD -- converted-ethereum-tests.txt | grep "^+" | grep -v "^+++" || true)
-          if [ -z "$lines" ]; then
-              echo "No new lines in converted-ethereum-tests.txt, check updates instead:"
+          source $GITHUB_ENV
+          new_sources=$(echo "$CHANGED_TEST_FILES" | tr ',' '\n')
+          files=$(uv run fill $new_sources --show-ported-from --clean --quiet --links-as-filled | grep .json)
+          if [[ -z "$files" ]]; then
+              echo "No ported fillers found, check updates instead:"
               echo "converted_skip=true" >> $GITHUB_ENV
               exit 0
           else
               echo "converted_skip=false" >> $GITHUB_ENV
           fi
 
-          files=$(echo "$lines" | grep -oP '(?<=\+).+\.json')
-          for file in $files; do
-              echo $file
-          done
           echo "----------------"
           echo "Discovered existing json tests that will be BASE files:"
 

converted-ethereum-tests.txt

@@ -19,7 +19,8 @@ The output behavior of `fill` has changed ([#1608](https://github.com/ethereum/e
 
 #### `fill`
 
-- ✨ Add `ported_from` test marker to track Python test cases that were converted from static fillers in [ethereum/tests](https://github.com/ethereum/tests) repository [#1590](https://github.com/ethereum/execution-spec-tests/pull/1590).
+- ✨ Add the `ported_from` test marker to track Python test cases that were converted from static fillers in [ethereum/tests](https://github.com/ethereum/tests) repository [#1590](https://github.com/ethereum/execution-spec-tests/pull/1590).
+- ✨ Add a new pytest plugin, `ported_tests`, that lists the static fillers and PRs from `ported_from` markers for use in the coverage Github Workflow [#1634](https://github.com/ethereum/execution-spec-tests/pull/1634).
 - 🔀 Refactor: Encapsulate `fill`'s fixture output options (`--output`, `--flat-output`, `--single-fixture-per-file`) into a `FixtureOutput` class ([#1471](https://github.com/ethereum/execution-spec-tests/pull/1471),[#1612](https://github.com/ethereum/execution-spec-tests/pull/1612)).
 - ✨ Don't warn about a "high Transaction gas_limit" for `zkevm` tests ([#1598](https://github.com/ethereum/execution-spec-tests/pull/1598)).
 - 🐞 `fill` no longer writes generated fixtures into an existing, non-empty output directory; it must now be empty or `--clean` must be used to delete it first ([#1608](https://github.com/ethereum/execution-spec-tests/pull/1608)).

docs/CHANGELOG.md

@@ -13,6 +13,7 @@ addopts =
     -p pytest_plugins.solc.solc
     -p pytest_plugins.filler.filler
     -p pytest_plugins.filler.static_filler
+    -p pytest_plugins.filler.ported_tests
     -p pytest_plugins.shared.execute_fill
     -p pytest_plugins.forks.forks
     -p pytest_plugins.eels_resolver

pytest.ini

@@ -203,6 +203,17 @@ def pytest_addoption(parser: pytest.Parser):
     )
 
 
+def is_help_or_collectonly_mode(config: pytest.Config) -> bool:
+    """Check if pytest is running in a help or collectonly mode."""
+    return (
+        config.getoption("markers")
+        or config.getoption("collectonly")
+        or config.getoption("markers")
+        or config.getoption("show_ported_from")
+        or config.getoption("links_as_filled")
+    )
+
+
 @pytest.hookimpl(tryfirst=True)
 def pytest_configure(config):
     """
@@ -226,7 +237,7 @@ def pytest_configure(config):
     # Initialize fixture output configuration
     config.fixture_output = FixtureOutput.from_config(config)
 
-    if config.option.collectonly:
+    if is_help_or_collectonly_mode(config):
         return
 
     try:
@@ -272,7 +283,7 @@ def pytest_configure(config):
 @pytest.hookimpl(trylast=True)
 def pytest_report_header(config: pytest.Config):
     """Add lines to pytest's console output header."""
-    if config.option.collectonly:
+    if is_help_or_collectonly_mode(config):
         return
     t8n_version = config.stash[metadata_key]["Tools"]["t8n"]
     return [(f"{t8n_version}")]
@@ -866,7 +877,7 @@ def pytest_sessionfinish(session: pytest.Session, exitstatus: int):
         return
 
     fixture_output = session.config.fixture_output  # type: ignore[attr-defined]
-    if fixture_output.is_stdout or session.config.option.collectonly:
+    if fixture_output.is_stdout or is_help_or_collectonly_mode(session.config):
         return
 
     # Remove any lock files that may have been created.