Have one clear API for calling `Sphinx` without unintended side-effects (deprecate `SphinxTestApp`?)

[chrisjsewell]

Currenty, there is no clearly documented way to "properly" use Sphinx programatically.
By "properly", I mean in a way that could be called multiple times within the same Python interpretor, without any unexpected polluting of the global state, and thus act exactly the same as if each call had been made in a separate Python interpretor

At present, there are diverging ways to do this, for the main CLI and for test cases:

with patch_docutils(confdir), docutils_namespace():
	app = Sphinx(...)
    app.build(...)

app = SphinxTestApp(...)
app.build(...)
app.cleanup()

Its not really clear from first appearance, but these actually almost achieve the same thing:

• docutils_namespace, on exit, resets the directives/roles lookups and additional node visitors
• SphinxTestApp.cleanup resets the directives/roles lookups and additional node visitors

However, there are also clear differences:

• patch_docutils temporarily patches docutils in a number of ways, that do not appear to be addressed by SphinxTestApp, in essence meaning you are not strictly testing the behaviour of Sphinx in production when using SphinxTestApp
• SphinxTestApp.cleanup does addition global state cleaning, restoring: sys.path, sphinx.locale.translators, and sphinx.pycode.ModuleAnalyzer.cache

I feel this divergence is unecessary, and ideally there would be one clear way to call sphinx as an API, perhaps something like:

with Sphinx.restore_global_state(confdir):
	app = Sphinx(...)
	app.build(...)

cc @picnixz and @jayaddison for discussion, as it relates to #12093 etc

---

An additional note (as I just mentioned in #12089 (comment))
is that it is not currently possible to temporarily turn-off colored logging output.
This is annoying for if you want to assert the content of the warning stream

In the CLI it can be turned off, but is then never reset to its previous state: https://github.com/chrisjsewell/sphinx/blob/fa290049515c38e68edda7e8c17be69b8793bb84/sphinx/cmd/build.py#L327

[pauleveritt]

I'll add something that I'm interested in: multiple calls to build, passing in filenames.

Use case: a high-performance reloading server that is very different to the sphinx-autobuild approach. Currently, calling build repeatedly on the same sphinx.application.Sphinx instance has globals etc. as mentioned in this ticket.

[webknjaz]

In the context of testing, it'd also be nice if Sphinx exposed a pytest fixture for this.

[webknjaz]

In the context of testing, it'd also be nice if Sphinx exposed a pytest fixture for this.

I recently discovered that https://www.sphinx-doc.org/en/master/extdev/testing.html exists and even attempted using it. One of the problems it has is that it's totally undocumented, and another is that the fixture names aren't made for use externally. A third-party project would make use of it by injecting app and redefining root_dir — both names are so generic that it's impossible to guess what these are w/o prior knowledge.

[picnixz]

One of the problems it has is that it's totally undocumented, and another is that the fixture names aren't made for use externally. A third-party project would make use of it by injecting app and redefining root_dir — both names are so generic that it's impossible to guess what these are w/o prior knowledge.

I started documenting it somewhere. I can try cherry-picking the documentation only if it could help though

[webknjaz]

@picnixz well, the docs would help. But I'd want the names to make sense outside the Sphinx upstream too...