plone.distribution tests are not found in coredev (all pytest tests?)

[mauritsvanrees]

With current buildout.coredev branch 6.1:

$ bin/test -s plone.distribution
Total: 0 tests, 0 failures, 0 errors, 0 skipped in 0.000 seconds.

Problem 1: where are the tests?

In bin/test I see:

'--test-path', '/Users/maurits/shared-eggs/cp313/plone.distribution-3.0.0b2-py3.13.egg',

Problem: this egg does not contain the tests.. If we want to be able to test all packages in combination, the Python distributions (source dist, wheel) must include the tests. And this can be tricky because the tests directory is in the top level folder instead of in src/plone/distribution/tests.

AFAIK this top level tests directory is recommended for pytest and it can simplify test/tox setup in the package. Is that right? If so, we would want this to work.

Actually, the source distribution does contain the tests, but the wheel not. This should be solvable, but it involves diving into how to specify data files. And that page says at the bottom:

Historically, setuptools by way of easy_install would encapsulate data files from the distribution into the egg (see the old docs). As eggs are deprecated and pip-based installs fall back to the platform-specific location for installing data files, there is no supported facility to reliably retrieve these resources.

Instead, the PyPA recommends that any data files you wish to be accessible at run time be included inside the package.

My conclusion: having a top level tests directory makes it hard to run the tests of all packages combined, like we do now in buildout.coredev.

Problem 2: zope-testrunner versus pytest

Let's see if it does work when we use a checkout of plone.distribution. The bin/test file then contains this:

'--test-path', '/Users/maurits/community/plone-coredev/6.1/src/plone.distribution/src',

This obviously does not find any tests, because they are in .../6.1/src/plone.distribution/tests, which is just outside the test path.

Fine, let's edit the file:

'--test-path', '/Users/maurits/community/plone-coredev/6.1/src/plone.distribution/tests',

Still:

$ bin/test -s plone.distribution
Total: 0 tests, 0 failures, 0 errors, 0 skipped in 0.000 seconds.
$ bin/test --all -vvv -s plone.distribution
Running tests at all levels
Total: 0 tests, 0 failures, 0 errors, 0 skipped in 0.000 seconds.
$ bin/test --all -vvv src/plone.distribution/tests/
Running tests at all levels
Total: 0 tests, 0 failures, 0 errors, 0 skipped in 0.000 seconds.

Here it seems the problem is that the zope testrunner does not find any pytest tests.

Maybe installing with pip/mxdev helps, see the readme? Not immediately:

.venv/bin/pytest src/plone.distribution/tests/
=========================================================================================================== test session starts ============================================================================================================
platform darwin -- Python 3.12.7, pytest-7.4.4, pluggy-1.5.0
rootdir: /Users/maurits/community/plone-coredev/6.1/src/plone.distribution
plugins: cov-5.0.0, zope.pytestlayer-8.2, plone-0.5.0
collected 54 items                                                                                                                                                                                                                         

src/plone.distribution/tests/api/test_api_distribution.py EEEEE                                                                                                                                                                      [  9%]
src/plone.distribution/tests/api/test_api_site.py EEE                                                                                                                                                                                [ 14%]
src/plone.distribution/tests/distributions/test_distributions_testing.py EEEEEEEEEE                                                                                                                                                  [ 33%]
src/plone.distribution/tests/services/test_services_site.py EEEEEEEEEEE                                                                                                                                                              [ 53%]
src/plone.distribution/tests/services/test_services_system.py EEE                                                                                                                                                                    [ 59%]
src/plone.distribution/tests/utils/test_utils_data.py ..                                                                                                                                                                             [ 62%]
src/plone.distribution/tests/utils/test_utils_request.py .......                                                                                                                                                                     [ 75%]
src/plone.distribution/tests/utils/test_utils_schema.py EEEEEEEEE                                                                                                                                                                    [ 92%]
src/plone.distribution/tests/utils/test_utils_validation.py EEEE    
...
9 passed, 45 errors in 9.73s

So: we have some rethinking to do on our test setup.

[gforcada]

I see a few options:

• plone.api style: allow and encourage pytest usage outside core, but not on core
• have a separate list of packages that are using pytest rather than zope.testrunner and run them both in parallel (either in Jenkins on GHA)
• convert all tests to use pytest (with gocept.pytestlayer) is a bit feasible, I'm not sure if the plone.distribution style of pytest and the one from gocept.pytestlayer work hand in hand though

The lowest effort is the first one, the hardest is the last one, but maybe the middle option might be the path forward? It complicates the test setup yet-a-bit-more, but we are already splitting the test suite with Robot tests anyway, not to mention Volto that is not tested at all on any package that triggers a jenkins job run, though it has been like that since the beginning and no one seem to be complaining about it anyway 😓

[mauritsvanrees]

For the moment the first option is fine: do not use pytest in core, meaning: Products.CMFPlone or anything below it. It is fine for core add-ons, so the packages like plone.distribution that live above Products.CMFPlone.

We can look into running pytest tests separately, the second option. That may involve installing everything with pip instead of buildout for this part. But then first we make sure that something like .venv/bin/pytest src/plone.distribution/tests/ in buildout.coredev actually works, instead of throwing errors like above.

[davisagli]

@mauritsvanrees I did an experiment adding pytest to the [robot] section of tests.cfg which generates scripts that include a PYTHONPATH with all the Plone eggs. With this I'm able to run bin/pytest src/plone.distribution/tests successfully.

diff --git a/tests.cfg b/tests.cfg
index a0956a4b5..359ae0d94 100644
--- a/tests.cfg
+++ b/tests.cfg
@@ -136,8 +136,10 @@ eggs =
     robotframework-selenium2screenshots
     plone.app.robotframework[reload,debug]
     robotframework-browser
+    pytest
 # We do not want to install all scripts from the test-eggs.
 scripts =
+    pytest
     ride
     robot
     robot-debug

Personally I think we should:

1. Explore whether we can put the tests inside the package so that we can run tests against installed distributions. From what I recall, there was some weirdness with tox if pytest was discovering the tests in one location (relative to the cwd) but imports were finding them in another (inside the tox-generated virtualenv). But I'm not sure if this applies to us.
2. Create a separate CI job (or jobs) to run tests for packages that use pytest, but don't expect all packages to migrate to it all at once.