Problems with leaking development information into the sdist package.

[jenstroeger]

Recently the pyproject.toml file has become the goto-source for configuring a package as well as the tools used to build that same package. As such, when building an sdist with flit those settings “leak” into the published package via that pyproject.toml file.¹

Splitting dev tools configurations — and I consider flit a dev tool separate from the published package! — into a separate file (e.g. develop.toml) works mostly except for flit:

• using flit --ini-file develop.toml build --setup-py --format sdist complains:

  Config error: Neither [project] nor [tool.flit.metadata] found in pyproject.toml
  

  because the actual package metadata remains in pyproject.toml whereas the flit tool configuration is stored in develop.toml. (Also note that incorrect file name in the error message!)
• using flit build --setup-py --format sdist generates an sdist package whose embedded pyproject.toml contains the flit configuration and I argue that’s an unnecessary “leak”.

I’m not sure how to go about solving this without modifying flit’s behavior. I think it’s sensible to use --ini-file to configure flit and still require pyproject.toml for the package metadata itself; or one could imagine a cmd line switch that removes non-metadata tables from the toml file. However, moving package metadata into develop.toml wouldn’t work well if I want to install my package in editable mode (which requires pyproject.toml).

I appreciate your suggestions…

Footnotes

1. For more details see issue https://github.com/jenstroeger/python-package-template/issues/951 and PR https://github.com/jenstroeger/python-package-template/pull/952. ↩

[takluyver]

In general, the config that Flit expects is what's needed for building a package, and thus belongs in pyproject.toml.

I see from the linked issue on your own repo that you're specifically thinking about development dependencies. I habitually put those in optional dependencies (and Flit allows some shortcuts with that), but that's not really what optional dependencies are designed for. There is a fairly recent spec for dependency groups which provides a better place for these, but the spec explicitly says it goes in pyproject.toml. I haven't done anything about that in Flit yet - I'm not sure if I need to - but if I do, I'd follow the spec and get it from pyproject.toml.

If you are really concerned about hiding development dependencies from sdists (I don't think it matters much), then probably your best option is to put them in requirements.txt files and use pip to install them, e.g. pip install -r docs/requirements.txt.

The --ini-file option is kind of a hangover from an early design direction that didn't pan out - as you might guess from the fact it still refers to INI 😉 . It should probably be removed. I guess that would be part of Flit 4.

[In case anyone is concerned by the issue title, leaking does not imply a security leak, as far as I'm aware - although you should be aware that publishing a Python package includes publishing anything in pyproject.toml. This is probably true whichever tool you use to build the package.]

[jenstroeger]

External data…

It’s not about external data required to build and run a package; my question and concern centers around configuration settings in the pyproject.toml file that are not required to build the wheel from source and to install & run it. Particulalry if people aren’t aware that the pyproject.toml file is published on PyPI and use it to store internal or sensitive data.

Getting philosophical 😁 ...

Actually I find the definition of “sdist” rather practical and relevant, because it defines what should and should not be published in an sdist. I think that the source distribution to publish a package and a development archive of a package are two different things. Above you said:

And I personally like sdists on PyPI being relatively complete, with stuff like tests and dev tool config present, because it serves as an archive of the project's history if its repository disappears one day.

Do folks really package tests into sdists? What about other build resources like Dockerfiles, Makefiles, etc. that are checked into a repository but that are not necessarily relevant to build the wheel? The “serves as an archive” actually sounds more like a git archive than a Python sdist packaging problem.

However, I think I need to read carefully through the packaging docs and related PEPs again…

[takluyver]

It’s not about external data required to build and run a package;

OK, but the tool.flit.external-data table that I'm talking about is about that.

Do folks really package tests into sdists?

Yes, absolutely, and I've had complaints multiple times where I've unwittingly left tests out of the sdist, mostly from Linux distro packagers who want to start with the sdist, prepare the package and run the tests. #703 is one fairly recent example. The source materials for building the docs are also generally expected to be there.

What about other build resources like Dockerfiles, Makefiles, etc. that are checked into a repository but that are not necessarily relevant to build the wheel?

It depends on the project and what these do, but I would often include them. E.g. to build the docs locally I would usually cd into the docs folder and run make html, so the Makefile is a necessary part of that.

The “serves as an archive” actually sounds more like a git archive than a Python sdist packaging problem.

I think a git archive is a pretty good starting point for an sdist, and historically Flit has tried to do roughly the same thing - and still does for now if you use flit build to make an sdist. But this turned out not to be a great fit for the packaging ecosystem, so I'm slowly moving away from that idea, back towards configuring the files to go in an sdist more manually.

[jenstroeger]

I'm slowly moving away from that idea, back towards configuring the files to go in an sdist more manually.

As in using the include and exclude options to stuff whatever we discussed above into the sdist?

That would actually shift the currently popular “build-from-source” conversation into an interesting direction if one was to stuff all that test and check cruft into an sdist as well… 🤔

[takluyver]

As in using the include and exclude options to stuff whatever we discussed above into the sdist?

Yup, exactly.

all that test and check cruft

Again, it's normal to put tests in an sdist, and people open issues if it's not there. One person's cruft is another person's useful data...

[jenstroeger]

Alright, let me do some homework. There’s a bunch of related discussion over in the packaging forum, namely

• PEP 625 – Filename of a Source Distribution (discussion)
• PEP 643 – Metadata for Package Source Distributions (discussion)
• Purpose of an sdist
• Should sdists include docs and tests?
• What do we want in standardized sdists?

and a few more.