Documenting a generic way to override pytest options

[ssbarnea]

While using xdist and enabling it by default inside tox.ini file is something a project is likely to do, there is a problem with it when it comes to development and ability to run/debug a single/subset of tests, case where you are likely not to want to use xdist.

One common mistake I seen people doing was to add xdist or coverage arguments to addops inside pytest.ini. The problem with this is that it kinda force you to run always with them, even when you may only want to debug a single test failure.

One workaround is to this is to use PYTEST_ADDOPS= pytest ... in order to prevent configuration options from being used. Still, this is inconvenient as it would be hard to remember the exact name of that var, not to mention that most people do not even know about its existence.

Another approach that I used in multiple projects was to do something like this in tox.ini files:

commands =
  {posargs:\
    -n0
    -p pytest_cov \
    --cov ansiblelint \
    --cov-report term-missing:skip-covered \
    --no-cov-on-fail}

This ensures that when you run tox without positional arguments you rely on the implicit set of arguments, but when giving any arguments, those will just replace the default ones instead of being added after these.

While working at https://github.com/ansible/ansible-navigator/pull/832/files I even consired adding {env:PYTEST_ADDOPTS} inside the default part of tox, ensuring that, when defined, it will take precedence over what is configured inside the tox.ini file. As Sviat noted on the review, that is bit against the specification of PYTEST_ADDOPS, which is documented as stuff that has less precedence than cli args.

# example 2
commands =
  {posargs:\
    -n0
    -p pytest_cov \
    --cov ansiblelint \
    --cov-report term-missing:skip-covered \
    --no-cov-on-fail
    {env:PYTEST_ADDOPTS}}

In my view I think that whatever default arguments are configured inside tox.ini file act as configuration, so repurposing PYTEST_ADDOPS to override those values would be acceptable, especially as it will not override real command line arguments passed to tox.

For example

• PYTEST_ADDOPS="-n1" tox -e py -- -n0 will still use -n0 as one may expect, while avoiding to load any default values from tox.ini
• PYTEST_ADDOPS="-n1" tox -e py -- would use -n1 because PYTEST_ADDOPS will override the initial -n0 from defaults.

The big question: is ok to use PYTEST_ADDOPS or we should use a different variable name for this purpose? If other name is recommended, which one would you suggest?

[nicoddemus]

Hi @ssbarnea,

The problem with this is that it kinda force you to run always with them, even when you may only want to debug a single test failure.

For this particular case, if you execute pytest -n0, the command line arguments will be appended to the command-line, so -n0 will be put after the ones defined in addopts, effectively overwriting it.

I understand the topic is more general, but thought I would mention this as this might be an immediate workaround for that problem in particular.

One workaround is to this is to use PYTEST_ADDOPS= pytest ... in order to prevent configuration options from being used.

Note that it doesn't actually prevent, it merely adds the options to the command line. This is the final arguments order as evaluated by pytest:

$PYTEST_ADDOPTS + config.addopts + command-line args

Here's the relevant code:

pytest/src/_pytest/config/__init__.py

Lines 1168 to 1180 in 90b1c93

	def _preparse(self, args: List[str], addopts: bool = True) -> None:
	if addopts:
	env_addopts = os.environ.get("PYTEST_ADDOPTS", "")
	if len(env_addopts):
	args[:] = (
	self._validate_args(shlex.split(env_addopts), "via PYTEST_ADDOPTS")
	+ args
	)
	self._initini(args)
	if addopts:
	args[:] = (
	self._validate_args(self.getini("addopts"), "via addopts config") + args
	)

EDIT: I was writing my answers as I read the issue, and by the end it is clear that the behavior I describe above is already understood by you, but decided to left it as it makes the behavior explicit for others reading this later.

is ok to use PYTEST_ADDOPS or we should use a different variable name for this purpose?

I think you should use a different name, to avoid confusion. PYTEST_ADDOPTS is already handled by pytest, trying to repurpose it implicitly inside tox.ini seems confusing.

If other name is recommended, which one would you suggest?

How about ANSIBLE_PYTEST_OPS? This is clearly a customization done by ansible, so there would be no mixup with pytest's own environment variables.

[ssbarnea]

Thanks Burno!

I would avoid using a variable that is specific to a project or organization as I plan to use this pattern in unrelated projects. I see no harm in using something like PYTEST_MYOPS (or PYTEST_FORCEOPS?), especially as it is likely the passenv includes PYTEST_* on those projects. I know the risk that it may clash with a future option of pytest but I can live with that, as it i more likely to be hit by a meteor.

Another reason why I asked about this is that I kinda hope that maybe we end-up documenting this somewhere in pytest docs, as way to use pytest with tox. Many people do not know exactly how to use those in tandem in an optimum or flexible way.