Find root cause of flaky test

[guettli]

I have a flaky test which failure is not reproducible for me.

def test_bar():
    ...
    foo = bar(blue)
    assert foo == 'sunshine'

Up to now I only see a meaningless error message of the test fails.

like moonlight != sunshine

I would like to trace bar() every time it is called in the test.

If the assertion fails, then I want to see the trace as text.

The trace should contain all lines which get executed by bar() (excluding Python stdlib).

Maybe something like this:

def test_bar():
    ...
    tracer = SomeMagicTracer()
    foo = tracer.call(bar, blue)
    assert foo == 'sunshine', tracer.trace_to_text()

How to get SomeMagicTracer and trace_to_text()?

[RonnyPfannschmidt]

https://docs.python.org/3/library/sys.html#sys.settrace

what to print is currently left as execise, maybe someone has already implemented a tracer at your desired detail level

[guettli]

I found a nice solution in this Stackoverflow answer: https://stackoverflow.com/a/68424316/633961

[1Mark]

While the answer linked above is nice, it would be really good if this was a fixture. This current solution forces you to
move the test into a method so it can be registered. Any suggestions would be appreciated.

The issue lies with ret = tracer.runfunc(f, *args, **kwargs)

def f(x):
    import random

    choice = random.choice((True, False))

    if choice:
        raise ValueError
    else:
        return x * 2


def trace(f, *args, **kwargs):
    import trace
    import sys

    tracer = trace.Trace(
        ignoredirs=(sys.base_exec_prefix, sys.base_prefix), timing=True,
    )
    ret = tracer.runfunc(f, *args, **kwargs)
    r = tracer.results()
    r.write_results(coverdir="/tmp/xx-trace")
    return ret


def test_f():
    with pytest.raises(ValueError):
        trace(f, 3)

[guettli]

But maybe a fixture is not the right approach. Because I only want to trace the function call. All lines called before and after the function call should not be traced.

[1Mark]

Hmm true, meanwhile it allows you to not have to update the test each time if it's an autouse fixture at the expense of some extra logging. I'd say it's more than worth it.

Especially since it's flaky and only happens during CI CD.

[guettli]

@1Mark I am deeply in a different topic, so I can't create a nice PR for a fixture. But if you want to do it, then this would be great.

For a quick solution it might make sense to update this page, so that more people know about this solution: https://docs.pytest.org/en/6.2.x/flaky.html