Pollution test cases

[cdce8p]

While debugging #1551, I used dectect-test-pollution to isolate the test case.

The tool does have a fuzzy mode which can find other pollution test cases. Unfortunately, it didn't even make it past round 1 😕
It seems like our test suite doesn't clean up properly and is somewhat reliant on a global state. For now it's fine to continue as is though, at some point, these issues might come back to haunt us.

Install

pip install detect-test-pollution

Run with (Python 3.10)

detect-test-pollution --tests tests/ --fuzz --runs 10

If someone would like to take a look, that would be greatly appreciated 🙏🏻

/CC: @jacobtylerwalls and @DanielNoord I saw you two worked on caching recently, maybe this is something you could take a look at?

/CC: @Pierre-Sassoulas

--

The first case I saw

Failing test: tests/unittest_brain.py::TestIsinstanceInference::test_isinstance_int_true
Pollution test: tests/unittest_manager.py::ClearCacheTest::test_brain_plugins_reloaded_after_clearing_cache

pytest \
    tests/unittest_manager.py::ClearCacheTest::test_brain_plugins_reloaded_after_clearing_cache \
    tests/unittest_brain.py::TestIsinstanceInference::test_isinstance_int_true 

[Pierre-Sassoulas]

It seems like our test suite doesn't clean up properly and is somewhat reliant on a global state.

That would be the astroid cache, I guess ? 😄 We can clean it up now but recently it was a mutable default value of a function. I don't think we added the cache clean function for each tests nor should we do it imo.

[jacobtylerwalls]

I found the cause. #1460 introduced a global state that can be wrong if the cache is cleared (and the builtins module is rebuilt). I have a tiny diff to fix it.