How should performance tests look like?

[MartinThoma]

The goal of performance tests is to discover catastrophic performance degradations (e.g. a critical loop becoming quadratic runtime from currently linear). It is not about benchmarking.

We want performance tests to be stable. They may not cause random timeouts.

We could use pytest-timeout (as a test decorator), time.time, or thread_time as shown in #2142

How is this done in other libraries?

Key Decisions

• The time to download PDFs should NOT be part of the performance test. Either the measurement needs to exclude it or the file needs to be downloaded before the test is executed.

[MartinThoma]

fpdf2 has a decorator @ensure_exec_time_below(seconds=10): https://github.com/py-pdf/fpdf2/blob/master/test/fonts/test_charmap.py#L40

From the usage it looks similar to the pytest-timeout decorator.

[pubpub-zz]

what I dislike with pytest-timeout is the name: a timeout is here to detect/prevent infinite loops : ensure_exec_time_below sound clearer
thread_time has 2 advantage for me :
a) less sensitive to global CPU loading that may slow down execution
b)really focus the time measurement of the code to be measured and be independant of setup (pdf loading...)

[MartinThoma]

really focus the time measurement of the code to be measured

To emphasize that point: We sometimes need setup code. Using the thread_time allows us to wrap only the code that we're interested in, ignoring any setup code

That is a pretty good argument to me 👍