Merge pull request #1976 from davidhewitt/pyo3-build-config-docs

docs: pyo3 config files

Author: davidhewitt

Architecture.md

@@ -28,8 +28,9 @@ To summarize, there are six main parts to the PyO3 codebase.
    - [`src/class`]
 5. [Procedural macros to simplify usage for users.](#5-procedural-macros-to-simplify-usage-for-users)
    - [`src/derive_utils.rs`], [`pyo3-macros`] and [`pyo3-macros-backend`]
-6. [`build.rs`](#6-buildrs)
+6. [`build.rs` and `pyo3-build-config`](#6-buildrs-and-pyo3-build-config)
    - [`build.rs`](https://github.com/PyO3/pyo3/tree/main/build.rs)
+   - [`pyo3-build-config`]
 
 ## 1. Low-level bindings of Python/C API
 
@@ -50,7 +51,7 @@ With `Py_LIMITED_API`, we can build a Python-version-agnostic binary called an
 `Py_37` means that the API is available from Python >= 3.7.
 There are also `Py_38`, `Py_39`, and so on.
 `PyPy` means that the API definition is for PyPy.
-Those flags are set in [`build.rs`](#6-buildrs).
+Those flags are set in [`build.rs`](#6-buildrs-and-pyo3-build-config).
 
 ## 2. Bindings to Python objects
 
@@ -166,25 +167,37 @@ some internal tricks for making `#[pyproto]` flexible.
 [`src/derive_utils.rs`] contains some utilities used in code generated by these proc-macros,
 such as parsing function arguments.
 
-## 6. `build.rs`
+## 6. `build.rs` and `pyo3-build-config`
 
-PyO3's [`build.rs`](https://github.com/PyO3/pyo3/tree/main/build.rs) is relatively long
-(about 900 lines) to support multiple architectures, interpreters, and usages.
-Below is a non-exhaustive list of its functionality:
+PyO3 supports a wide range of OSes, interpreters and use cases. The correct environment must be
+detected at build time in order to set up relevant conditional compilation correctly. This logic
+is captured in the [`pyo3-build-config`] crate, which is a `build-dependency` of `pyo3` and
+`pyo3-macros`, and can also be used by downstream users in the same way.
 
-- Cross-compiling support.
-  - If `TARGET` architecture and `HOST` architecture differ, we find cross compile information
-    from environment variables (`PYO3_CROSS_LIB_DIR` and `PYO3_CROSS_PYTHON_VERSION`) or system files.
+In [`pyo3-build-config`]'s `build.rs` the build environment is detected and inlined into the crate
+as a "config file". This works in all cases except for cross-compiling, where it is necessary to
+capture this from the `pyo3` `build.rs` to get some extra environment variables that Cargo doesn't
+set for build dependencies.
+
+The `pyo3` `build.rs` also runs some safety checks such as ensuring the Python version detected is
+actually supported.
+
+Some of the functionality of `pyo3-build-config`:
 - Find the interpreter for build and detect the Python version.
-  - We have to set some version flags like `Py_37`.
-  - If the interpreter is PyPy, we set `PyPy`.
-  - If `PYO3_NO_PYTHON` environment variable is set then the interpreter detection is bypassed
+  - We have to set some version flags like `#[cfg(Py_3_7)]`.
+  - If the interpreter is PyPy, we set `#[cfg(PyPy)`.
+  - If the `PYO3_CONFIG_FILE` environment variable is set then that file's contents will be used
+    instead of any detected configuration.
+  - If the `PYO3_NO_PYTHON` environment variable is set then the interpreter detection is bypassed
     entirely and only abi3 extensions can be built.
 - Check if we are building a Python extension.
   - If we are building an extension (e.g., Python library installable by `pip`),
     we don't link `libpython`.
     Currently we use the `extension-module` feature for this purpose. This may change in the future.
     See [#1123](https://github.com/PyO3/pyo3/pull/1123).
+- Cross-compiling configuration
+  - If `TARGET` architecture and `HOST` architecture differ, we find cross compile information
+    from environment variables (`PYO3_CROSS_LIB_DIR` and `PYO3_CROSS_PYTHON_VERSION`) or system files.
 
 <!-- External Links -->
 
@@ -194,6 +207,7 @@ Below is a non-exhaustive list of its functionality:
 
 [`pyo3-macros`]: https://github.com/PyO3/pyo3/tree/main/pyo3-macros
 [`pyo3-macros-backend`]: https://github.com/PyO3/pyo3/tree/main/pyo3-macros-backend
+[`pyo3-build-config`]: https://github.com/PyO3/pyo3/tree/main/pyo3-build-config
 
 <!-- Directories -->
 

guide/pyo3_version.py

@@ -18,11 +18,13 @@
 
 if PYO3_VERSION_TAG == "main":
     PYO3_DOCS_URL = "https://pyo3.rs/main/doc"
+    PYO3_DOCS_VERSION = "latest"
     PYO3_CRATE_VERSION = 'git = "https://github.com/pyo3/pyo3"'
 else:
     # v0.13.2 -> 0.13.2
     version = PYO3_VERSION_TAG.lstrip("v")
     PYO3_DOCS_URL = f"https://docs.rs/pyo3/{version}"
+    PYO3_DOCS_VERSION = version
     PYO3_CRATE_VERSION = f'version = "{version}"'
 
 
@@ -35,6 +37,7 @@ def replace_section_content(section):
         section["Chapter"]["content"]
         .replace("{{#PYO3_VERSION_TAG}}", PYO3_VERSION_TAG)
         .replace("{{#PYO3_DOCS_URL}}", PYO3_DOCS_URL)
+        .replace("{{#PYO3_DOCS_VERSION}}", PYO3_DOCS_VERSION)
         .replace("{{#PYO3_CRATE_VERSION}}", PYO3_CRATE_VERSION)
     )
 

guide/src/building_and_distribution.md

@@ -44,7 +44,11 @@ Caused by:
   build_flags=WITH_THREAD
 ```
 
-> Note: if you save the output config to a file, it is possible to manually override the contents and feed it back into PyO3 using the `PYO3_CONFIG_FILE` env var. For now, this is an advanced feature that should not be needed for most users. The format of the config file and its contents are deliberately unstable and undocumented. If you have a production use-case for this config file, please file an issue and help us stabilize it!
+### Advanced: config files
+
+If you save the above output config from `PYO3_PRINT_CONFIG` to a file, it is possible to manually override the contents and feed it back into PyO3 using the `PYO3_CONFIG_FILE` env var.
+
+If your build environment is unusual enough that PyO3's regular configuration detection doesn't work, using a config file like this will give you the flexibility to make PyO3 work for you. To see the full set of options supported, see the documentation for the [`InterpreterConfig` struct](https://docs.rs/pyo3-build-config/{{#PYO3_DOCS_VERSION}}/pyo3_build_config/struct.InterpreterConfig.html).
 
 ## Building Python extension modules