tests: improve the sbi test suite

[dgedon]

Currently, we have a great set of tests which take a considerable amount of time for PRs. Potentially we want to mark some more tests as "slow" for the hackathon to ensure faster feedback during development. The goal would be to prioritize core functionality tests for PR validation while allowing more time-intensive tests to run separately at a later stage.

Update by @janfb : Please see general info in comment below.

[janfb]

Yes, very good point. Before the hackathon, we should

1. further reduce the set of core tests to make the CI faster. We should then make sure to CD on the PR branch once before merging into main

2. find ways to make the test suite more efficient, e.g., by using fixtures for code snippets that are repeated across a test module, e.g., use a pytest fixture that trains NPE on a linear Gaussian task and then re-use this trained posterior across the linear_gaussian_snpe_test.py module where possible?

[janfb]

another idea would be caching test results across CI runs within a feature branch and then executing only the previously failed tests with every new push:

something like:

    - name: Cache pytest results
      uses: actions/cache@v4
      with:
        path: .pytest_cache
        key: ${{ runner.os }}-pytest-${{ github.ref }}
        restore-keys: |
          ${{ runner.os }}-pytest-

    - name: Run only failing tests from last CI run
      run: |
        pytest --lf -v -x -n auto -m "not slow and not gpu" --cov=sbi --cov-report=xml tests/ || \
        pytest -v -x -n auto -m "not slow and not gpu" --cov=sbi --cov-report=xml tests/

note the || between the pytest calls, which will execute all tests only if the any recently failed tests fail again.

NOTE: this seems a bit hacky and should be tested carefully!

[janfb]

yet another idea to speed up CI would be to call only the tests of the changed files, e.g., using https://testmon.org/

this could look something like:

• collect tests affected by the current files changes into .testmondata
• cache this file across CI runs
• run pytest with testmon option

- name: Cache testmon data
  uses: actions/cache@v4
  with:
    path: .testmondata
    key: testmon-${{ github.ref }}-${{ github.sha }}
    restore-keys: |
      testmon-${{ github.ref }}-
      testmon-

- name: Run the fast CPU tests with testmon
  run: |
    pytest --testmon -v -x -n auto -m "not slow and not gpu" --cov=sbi --cov-report=xml tests/

[janfb]

Update:

General info: we have a large test suite with some tests marked as slow. We have a Continuous Integration workflow that runs only the "not slow" tests and is executed for every new push to a feature branch / pull request. In addition, we have Continuous Deployment workflow that runs all tests including the "slow" tests and is executed with every merge into main.

We have already implemented some changes to CI already:

• we are now using uv to install dependencies, which is much faster than using pip.
• we use a pytest plugin testmon to keep track of which files changed in a new PR commit and to re-run only the corresponding tests.

Thus, at the moment CI takes around 40min for a full run, but can take much less time in subsequent pushes to the PR because of testmont.

CD takes around 2 hourse at the moment.

This issue will have a couple sub-issues to improve the test suite.

[vagechirkov]

Now that we have working regression tests, we can significantly speed up testing without compromising coverage (see #1461). Thanks @schroedk! This could be an easy improvement for tests that are slow but not marked as slow because they are critical. I think it would be great to collect the list of the next candidates for similar improvement. Any suggestions for such tests?