Skip to content

Conversation

evnchn
Copy link
Collaborator

@evnchn evnchn commented Oct 8, 2025

Motivation

#5232 (make Mermaid tests not flaky) harder than it looks. We may have to live with flakiness. This PR does exactly that.

Implementation

This pull request introduces improvements to the test suite's reliability by addressing flaky tests related to Mermaid rendering. The main changes involve adding the pytest-rerunfailures plugin and marking relevant tests to automatically rerun on failure, reducing false negatives in CI.

Test reliability improvements:

  • Added pytest-rerunfailures as a dependency in both pyproject.toml and tests/requirements.txt to enable automatic rerunning of flaky tests. [1] [2]
  • Marked Mermaid-related tests in tests/test_markdown.py with @pytest.mark.flaky(reruns=2) to allow up to two reruns on failure. [1] [2]
  • Applied a module-level pytest.mark.flaky(reruns=2) marker to all tests in tests/test_mermaid.py for consistent rerun behavior.
  • Imported pytest in relevant test files to support the new flaky test markers.

Progress

  • I chose a meaningful title that completes the sentence: "If applied, this PR will..."
  • The implementation is complete.
  • Pytests have been added (or are not necessary).
  • Documentation has been added (or is not necessary).

@evnchn evnchn added the testing Type/scope: Pytests and test fixtures label Oct 8, 2025
@evnchn
Copy link
Collaborator Author

evnchn commented Oct 10, 2025

OK so with the photos we verified that Mermaid is indeed not working when the tests fail. As in #5232 (comment)

If that is the case, this PR is not productive as it shoves problem under a blanket.

Re-opening it is welcome if we determined that the problem does not affects end users.

@evnchn evnchn closed this Oct 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing Type/scope: Pytests and test fixtures

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant