Skip to content

Commit 54981ce

Browse files
lwasserlwasser
committed
Update notes-from-review.md
1 parent b937bc4 commit 54981ce

File tree

1 file changed

+0
-312
lines changed

1 file changed

+0
-312
lines changed

‎package-structure-code/notes-from-review.md

Lines changed: 0 additions & 312 deletions
Original file line numberDiff line numberDiff line change
@@ -1,313 +1 @@
1-
<!-- ### Build tools for Python packages with complex build steps
2-
If your package is not pure Python, or it has complex build steps (or build
3-
steps that you need to customize), then you should consider using:
41

5-
* Setuptools
6-
* Hatch
7-
* PDM
8-
9-
These tools allow you to customize your workflow with build steps needed
10-
to compile code. -->
11-
12-
<!-- From stefan: build, run tests on built version, load the built version into
13-
Python (?how is this different from install??), make editable install, build
14-
wheel, build sdist -->
15-
16-
<!--
17-
The example below is taken from [this thread in GitHub](https://github.com/py-pkgs/py-pkgs/issues/95#issuecomment-1035584750).
18-
19-
```toml
20-
[tool.poetry.dependencies]
21-
python = ">=3.6" # This is muddled in as a dependency, while it's not like the others
22-
numpy = ">=1.13.3"
23-
typing_extensions = { version = ">=3.7", python = "<3.8" }
24-
25-
sphinx = {version = "^4.0", optional = true}
26-
sphinx_book_theme = { version = ">=0.0.40", optional = true }
27-
sphinx_copybutton = { version = ">=0.3.1", optional = true }
28-
pytest = { version = ">=6", optional = true }
29-
importlib_metadata = { version = ">=1.0", optional = true, python = "<3.8" } # TOML error to add an ending comma or new line, even if this gets long
30-
boost-histogram = { version = ">=1.0", optional = true }
31-
32-
[tool.poetry.dev-dependencies]
33-
pytest = ">=5.2" # All the optional stuff above doesn't help here!
34-
importlib_metadata = {version = ">=1.0", python = "<3.8" }
35-
boost_histogram = ">=1.0"
36-
37-
[tool.poetry.extras]
38-
docs = ["sphinx", "sphinx_book_theme", "sphinx_copybutton"]
39-
test = ["pytest", "importlib_metadata", "boost_histogram" ]
40-
```
41-
42-
vs PDM
43-
44-
```toml
45-
[project]
46-
requires-python = ">=3.6"
47-
dependencies = [
48-
"numpy>=1.13.3",
49-
"typing-extensions>=3.7; python_version<'3.8'",
50-
]
51-
52-
# Only needed for extras
53-
[project.optional-dependencies]
54-
docs = [
55-
"sphinx>=4.0",
56-
"sphinx-book-theme>=0.0.40",
57-
"sphinx-copybutton>=0.3.1",
58-
]
59-
test = [
60-
"pytest>=6",
61-
"importlib-metadata>=1.0; python_version<'3.8'",
62-
"boost-histogram>=1.0",
63-
]
64-
65-
# Only needed for "dev" installs
66-
[tool.pdm.dev-dependencies]
67-
dev = [
68-
"pytest>=6",
69-
"importlib-metadata>=1.0; python_version<'3.8'",
70-
"boost-histogram>=1.0",
71-
]
72-
```
73-
74-
From Eli:
75-
76-
poetry: supports it (c extensions), but is undocumented and uses setuptools under the hood, they plan to change how this works and then document it
77-
pdm-back-end: supports it, and documents it -- and also uses setuptools under the hood
78-
hatchling: permits you to define hooks for you to write your own custom build steps, including to build C++ extensions
79-
80-
81-
**PDM** does have some support for `C`/[`Cython`](https://cython.org/) extensions. [Click here to
82-
learn more.](https://pdm.fming.dev/latest/pyproject/build/#build-platform-specific-wheels). This functionality uses setuptools "under the
83-
hood".
84-
85-
86-
-->
87-
88-
<!--
89-
### Build front-ends
90-
91-
Build front-ends have a user-friendly interface that allow you to perform
92-
common Python packaging tasks such as building your package, creating an
93-
environment to run package tests and build documentation, and pushing to PyPI.
94-
95-
For instance, you can use **Flit**, **Hatch**, **Poetry** and **PDM** to both build your
96-
package and to publish your package to PyPI (or test PyPI). However, if you
97-
want a tool that also support environment management and versioning your package,
98-
then you might prefer to use **Hatch**, **Poetry** or **PDM**.
99-
100-
Using a tool like **Flit**, **Hatch**, **Poetry** or **PDM** will simplify your workflow.
101-
102-
Example to build your package with **Flit**:
103-
104-
`flit build`
105-
106-
Example to publish to PyPI:
107-
`flit publish --repository testpypi`
108-
109-
In the Python package build space **setuptools** is
110-
the "OG" -the original tool that everyone used to use.
111-
With a tool like `setuptools` you have the flexibility
112-
to publish python pure python packages and packages with custom build steps. However, you will also need to use other tools. For example, you will use `twine` to publish to PyPI.
113-
114-
## An ecosystem of Python build tools
115-
116-
Below we introduce several of the most commonly used
117-
Python packaging build tools. Each tool has various
118-
features that might make you chose to use it
119-
or not use it. There is no right or wrong tool to use
120-
as far as pyOpenSci is concerned. We are just trying to
121-
help you find the tool that works best for
122-
your workflow.
123-
Example build steps using setuptools:
124-
======= -->
125-
126-
<!-- TODO: create build tool selection diagram - https://www.canva.com/design/DAFawXrierc/O7DTnqW5fmMEZ31ao-TK9w/edit -->
127-
128-
<!--
129-
On this page, we will focus on using front-end tools to package pure python
130-
packages. We will note if a package does have the flexibility to support other
131-
back-ends and in turn more complex builds (*mentioned in #2 and #3 above*). -->
132-
<!--
133-
## COmbine the two sets of statement below...
134-
ELI:
135-
PDM supports C/Cython extensions too: https://pdm.fming.dev/latest/pyproject/build/#build-platform-specific-wheels
136-
137-
It does this by allowing you to write a python script that gets injected into a setuptools build process :) so that's not necessarily the greatest choice. It's a bit like using setuptools directly. ;)
138-
139-
Ralf:
140-
Hatch only supports pure Python packages as of now. setuptools is still a very reasonable choice, and okay if all you have is a few C/Cython extensions. But I'd say you should probably recommend meson-python and scikit-build-core as the two best tools for building packages containing compiled extensions.
141-
142-
143-
* link to ralf's blog and book on complex builds
144-
* keep this page high level so we don't get weight downsides
145-
* can use the examplePy repo stefan and I are working on that will test various build combinations
146-
147-
*****
148-
149-
ELI: It would be more accurate to say that PDM supports using PDM and setuptools at the same time, so you run setuptools to produce the C extensions and then PDM receives the compiled extension files (.so, .pyd) and packages it up alongside the pure python files.
150-
151-
Hatch - https://hatch.pypa.io/latest/config/build/#build-hooks uild hooks
152-
153-
Ralf -
154-
Hatch has the worst take on building compiled code by some distance. Unless its author starts developing an understanding of build systems / needs, and implements support for PEP 517 build back-end hooks in pyproject.toml, it's pretty much a dead end.
155-
****
156-
157-
158-
HEnry: Poetry will move to PEP 621 configuration in version 2.
159-
160-
* pdm, hatch and poetry all have "ways" of supporting c extensions via pdm-backend, hatchling and poetry's build back-end.
161-
* poetry's support for C extensions is not fully developed and documented (yet). * Poetry doesn't offer a way to facilitate "communication" between poetry front end and another back-end like meson to build via a build hook. so while some have used it with other back-end builds it's not ideal for this application
162-
* pdm and poetry both rely on setuptools for C extensions. pdm's support claims to be fully developed and documented. poetry claims nothing, and doesn't document it.
163-
* hatch both offers a plugin type approach to support custom build steps
164-
PDM (right now) is the only tool that supports other back-ends (hatch is working on this - 2 minor releases away)
165-
At some point a build becomes so complex that you need to use a tool like scikit or meson to support that complexity.
166-
167-
168-
169-
**Setuptools** is the oldest tool in the above list. While it doesn't have a
170-
friendly user front end, because "OG" tool that has been used for Python packaging for over a decade, we discuss it here.
171-
172-
**Hatch** and PDM are newer, more modern tool that support customization of any
173-
part of your packaging steps. These tools also support some C and C++
174-
extensions.
175-
176-
177-
OFEK - Why use hatchlin vs pdm back-end -
178-
File inclusion is more configurable and easier by default
179-
There is already a rich ecosystem of plugins and a well-thought-out interface
180-
Consistency since the official Python packaging tutorial uses Hatchling by default
181-
182-
183-
Henry -
184-
The scikit-hep cookie provides 11 back-ends including flit-core and hatchling, and I've moved packaging to flit-core, and lots of other things to hatchling, and I can say that hatching's defaults are much nicer than flit-core's. Hatching uses .gitignore to decide what to put in the sdist. Flit-core basically tries to keep its hands off of adding defaults, so you have to configure everything manually. To make it even more confusing, if you use flit instead of a standard tool like build, it will switch to using VCS and those ignored files won't be added - meaning it is really easy to have a project that doesn't support build, including various GitHub Actions. Hatchling wins this by a ton.
185-
186-
<!-- TODO: add - compatible with other build back-ends eg pdm can work with hatchling
187-
188-
Eli:
189-
poetry: supports it, but is undocumented and uses setuptools under the hood, they plan to change how this works and then document it
190-
pdm-back-end: supports it, and documents it -- and also uses setuptools under the hood
191-
hatchling: permits you to define hooks for you to write your own custom build steps, including to build C++ extensions
192-
193-
-->
194-
195-
<!-- from eli about pdm
196-
It would be more accurate to say that PDM supports using PDM and setuptools at the same time, so you run setuptools to produce the C extensions and then PDM receives the compiled extension files (.so, .pyd) and packages it up alongside the pure Python files.
197-
198-
Comment about hatch.
199-
https://github.com/pyOpenSci/python-package-guide/pull/23#discussion_r1081108118
200-
201-
From ralf: There are no silver bullets here yet, no workflow tool is complete. Both Hatch and PDM are single-author tools, which is another concern. @eli-schwartz's assessment is unfortunately correct here I believe (at a high level at least, not sure about details). Hatch has the worst take on building compiled code by some distance. Unless its author starts developing an understanding of build systems / needs, and implements support for PEP 517 build back-end hooks in pyproject.toml, it's pretty much a dead end.
202-
203-
-->
204-
205-
<!--TODO Add examples of builds using each of the tools below?
206-
207-
pdm, hatch and poetry all have "ways" of supporting c extensions via pdm-build, hatchling and poetry's build back-end.
208-
poetry's support for C extensions is not fully developed and documented (yet). Poetry doesn't offer a way to facilitate "communication" between poetry front end and another back-end like meson to build via a build hook.
209-
PDM and hatch both offer a plugin type approach to support custom build steps
210-
PDM (right now) is the only tool that supports other back-ends (hatch is working on this - 2 minor releases away)
211-
At some point a build becomes so complex that you need to use a tool like scikit or meson to support that complexity.
212-
213-
CORRECTIONS:
214-
pdm doesn't use plugins. Hatch does.
215-
pdm and poetry both rely on setuptools for C extensions. pdm's support claims to be fully developed and documented. poetry claims nothing, and doesn't document it.
216-
217-
218-
??
219-
Poetry supports extensions written in other languages but this functionality is
220-
currently undocumented.
221-
222-
Tools such as Setuptools, PDM, Hatch and Poetry all have some level of support
223-
for C and C++ extensions.
224-
Some Python packaging tools,
225-
such as **Flit** and the **flit-core** build back-end only support pure-Python
226-
package builds.
227-
Some front-end packaging tools, such as PDM, allow you to use other
228-
build back-ends such as **meson** and **scikit-build**.
229-
230-
231-
me:
232-
pdm, hatch and poetry all have "ways" of supporting c extensions via pdm-build, hatchling and poetry's build back-end.
233-
poetry's support for C extensions is not fully developed and documented (yet). Poetry doesn't offer a way to facilitate "communication" between poetry front end and another back-end like meson to build via a build hook.
234-
PDM and hatch both offer a plugin type approach to support custom build steps
235-
PDM (right now) is the only tool that supports other back-ends (hatch is working on this - 2 minor releases away)
236-
At some point a build becomes so complex that you need to use a tool like scikit or meson to support that complexity.
237-
@eli-schwartz eli-schwartz 3 weeks ago
238-
PDM and hatch both offer a plugin type approach to support custom build steps
239-
240-
ELI:
241-
pdm doesn't use plugins. Hatch does.
242-
pdm and poetry both rely on setuptools for C extensions. pdm's support claims to be fully developed and documented. poetry claims nothing, and doesn't document it.
243-
244-
245-
https://pdm.fming.dev/latest/pyproject/build/#build-platform-specific-wheels
246-
-->
247-
248-
<!-- https://github.com/pyOpenSci/python-package-guide/pull/23#discussion_r1071541329
249-
ELI: A complex build could mean running a python script that processes some data file and produces a pure python module file.
250-
251-
Probably not common in the scientific community specifically, but I've seen quite a few setup.py files that contain custom build stages which e.g. build gettext locale catalogs.
252-
253-
The main point is that it is more "complex" than simply copying files or directories as-is into the built wheel.
254-
-->
255-
256-
<!--
257-
COMMENTED OUT TEXT TO BE MOVED
258-
259-
260-
# TODO LINK TO CI BUILDS FOR Documentation>
261-
Maybe we can curate a list of CI builds that people can use??? or is that moving too close to a cookie cutter situation
262-
263-
The text below is being moved to the packaging infrastructure section which
264-
doesn't exist YET... but will soon .
265-
pyOpenSci packages must:
266-
267-
- Contain full documentation for any user-facing functions.
268-
- Have a test suite that covers the major functionality of the package.
269-
- Use continuous integration.
270-
- Use an OSI approved software license.
271-
272-
273-
## Other recommendations
274-
### Python version support
275-
You should always be explicit about which versions of Python your package supports.
276-
Keeping compatibility with old Python versions can be difficult as functionality changes.
277-
A good rule of thumb is that the package should support, at least,
278-
the latest three Python versions (e.g., 3.8, 3.7, 3.6).
279-
280-
### Code Style
281-
pyOpenSci encourages authors to consult [PEP 8](https://www.python.org/dev/peps/pep-0008/) for information on how to style your code.
282-
283-
### Linting
284-
An automatic linter (e.g. flake8) can help ensure your code is clean and free of syntax errors. These can be integrated with your CI.
285-
286-
-->
287-
288-
<!--
289-
```{tip}
290-
### Python packaging resources that we love
291-
292-
We think the resources below are excellent but each have particular opinions
293-
that you may or may not find in our packaging guide. For instance, the PyPA
294-
guide encourages users to store their package in a `src/package-name` directory.
295-
While we accept that approach many of our community members prefer to not use
296-
the `src` directory.
297-
298-
* [Python packaging for research software engineers](https://merely-useful.tech/py-rse/)
299-
* [PyPA packaging guide](https://packaging.python.org/en/latest/)
300-
```
301-
-->
302-
303-
<!--
304-
* one of the benefits of wheel is pretty much avoiding setup.py which
305-
has code mixed in. makes you more vulnerable to a code injection on install.
306-
307-
assuming this means if the package is already pre-built than setup.py isn't running anything on install because install is just moving files across to the machine to be run.
308-
309-
And having metadata separate allows someone to view the metadata without
310-
running any python code as it's a machine and human readable format.
311-
312-
https://scikit-hep.org/developer/pep621
313-
-->

0 commit comments

Comments
 (0)