What hooks can throw errors?

[brandon-leapyear]

The documentation states:

Note that hook functions other than pytest_runtest_* are not allowed to raise exceptions. Doing so will break the pytest run.

This makes sense, where "break the pytest run" means "prevent all tests from running". But it seems like excluding pytest_runtest_* hooks, there are two different kinds of hooks: hooks where thrown errors result in an "INTERNALERROR" and hooks where thrown errors result in a normal one-time "ERROR".

For example, I'm trying to write a hook that runs after collecting all tests and throws an error if some condition fails. Throwing an exception in pytest_collection_modifyitems results in INTERNALERROR> Exception: ..., whereas throwing an exception in pytest_generate_tests results in ERROR collecting test/test_foo.py.

• Is the fact that pytest_collection_modifyitems and pytest_generate_tests have these differences in error handling guaranteed? i.e. if I throw an error in pytest_generate_tests, will some future pytest version change it to an INTERNALERROR?
• Is there an official list of hooks specifying whether a hook throws an INTERNALERROR or some other error?
• If I want a test run to result in an "error during collection", is pytest_generate_test the correct location?

[kmiller96]

Bumping this. I really want this functionality!