Skip to content

Sporadic failures with parallel builds through maturin #318

New issue
New issue
Open
#319
Open
Sporadic failures with parallel builds through maturin#318
#319
@tuxu

Description

@tuxu
tuxu
opened on Feb 12, 2025

I ran into a race condition with cargo-zigbuild's generated linker script when multiple such builds are driven in parallel through maturin and uv. I initially blamed uv, because the error message indicated an issue with the lifetime of the temporary build venv, however @konstin pointed out that the linker script generation here is not safe (astral-sh/uv#11444 (comment)).

Steps to reproduce: 
# Running inside a ubuntu:22.04 container with uv / maturin / zig
# CPU is Intel i7-13700K
root@23be7dfc1db1:/tmp/foo# uv --version
uv 0.5.29
root@23be7dfc1db1:/tmp/foo# maturin --version
maturin 1.8.2
root@23be7dfc1db1:/tmp# maturin new foo -b pyo3
  ✨ Done! New project created foo
root@23be7dfc1db1:/tmp# cd foo
root@23be7dfc1db1:/tmp/foo# maturin new bar -b pyo3
  ✨ Done! New project created bar
root@23be7dfc1db1:/tmp/foo# maturin new baz -b pyo3
  ✨ Done! New project created baz
root@23be7dfc1db1:/tmp/foo# echo -e "[tool.uv.workspace]\nmembers = [\"bar\", \"baz\"]\n" >> pyproject.toml
root@23be7dfc1db1:/tmp/foo# iter=0; while uv build --wheel --all-packages --out-dir build --config-setting "build-args=--zig" ; do ((iter++)); echo "iter: $iter" ; done
[bar] Building wheel...
[baz] Building wheel...
[foo] Building wheel...
Running `maturin pep517 build-wheel -i /root/.cache/uv/builds-v0/.tmpNCK6Wi/bin/python --compatibility off --zig`
Running `maturin pep517 build-wheel -i /root/.cache/uv/builds-v0/.tmpVvuph8/bin/python --compatibility off --zig`
    Updating crates.io index
    Blocking waiting for file lock on package cache
Running `maturin pep517 build-wheel -i /root/.cache/uv/builds-v0/.tmp3joi2Z/bin/python --compatibility off --zig`
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Updating crates.io index
    Blocking waiting for file lock on package cache
    Updating crates.io index
    Blocking waiting for file lock on package cache
     Locking 19 packages to latest compatible versions
    Blocking waiting for file lock on package cache
     Locking 19 packages to latest compatible versions
    Blocking waiting for file lock on package cache
     Locking 19 packages to latest compatible versions
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
🔗 Found pyo3 bindings
🔗 Found pyo3 bindings
🔗 Found pyo3 bindings
🐍 Found CPython 3.9 at /root/.cache/uv/builds-v0/.tmpNCK6Wi/bin/python
📡 Using build options features from pyproject.toml
🐍 Found CPython 3.9 at /root/.cache/uv/builds-v0/.tmp3joi2Z/bin/python
📡 Using build options features from pyproject.toml
🐍 Found CPython 3.9 at /root/.cache/uv/builds-v0/.tmpVvuph8/bin/python
📡 Using build options features from pyproject.toml
    Blocking waiting for file lock on package cache
   Compiling target-lexicon v0.12.16
   Compiling once_cell v1.20.3
   Compiling proc-macro2 v1.0.93
   Compiling unicode-ident v1.0.16
   Compiling libc v0.2.169
   Compiling autocfg v1.4.0
   Compiling heck v0.5.0
   Compiling cfg-if v1.0.0
   Compiling indoc v2.0.5
   Compiling unindent v0.2.3
    Blocking waiting for file lock on package cache
   Compiling target-lexicon v0.12.16
   Compiling once_cell v1.20.3
   Compiling proc-macro2 v1.0.93
   Compiling unicode-ident v1.0.16
   Compiling autocfg v1.4.0
   Compiling libc v0.2.169
   Compiling heck v0.5.0
   Compiling target-lexicon v0.12.16
   Compiling cfg-if v1.0.0
   Compiling once_cell v1.20.3
   Compiling proc-macro2 v1.0.93
   Compiling unicode-ident v1.0.16
   Compiling libc v0.2.169
   Compiling autocfg v1.4.0
   Compiling indoc v2.0.5
   Compiling heck v0.5.0
   Compiling unindent v0.2.3
   Compiling indoc v2.0.5
   Compiling unindent v0.2.3
   Compiling cfg-if v1.0.0
   Compiling memoffset v0.9.1
   Compiling memoffset v0.9.1
   Compiling memoffset v0.9.1
   Compiling pyo3-build-config v0.23.4
   Compiling pyo3-build-config v0.23.4
   Compiling quote v1.0.38
   Compiling pyo3-build-config v0.23.4
   Compiling quote v1.0.38
   Compiling quote v1.0.38
   Compiling syn v2.0.98
   Compiling syn v2.0.98
   Compiling syn v2.0.98
   Compiling pyo3-macros-backend v0.23.4
   Compiling pyo3-ffi v0.23.4
   Compiling pyo3 v0.23.4
   Compiling pyo3-ffi v0.23.4
   Compiling pyo3-macros-backend v0.23.4
   Compiling pyo3 v0.23.4
   Compiling pyo3-macros-backend v0.23.4
   Compiling pyo3-ffi v0.23.4
   Compiling pyo3 v0.23.4
   Compiling pyo3-macros v0.23.4
   Compiling pyo3-macros v0.23.4
   Compiling pyo3-macros v0.23.4
   Compiling bar v0.1.0 (/tmp/foo/bar)
   Compiling baz v0.1.0 (/tmp/foo/baz)
   Compiling foo v0.1.0 (/tmp/foo)
    Finished `release` profile [optimized] target(s) in 4.99s
📦 Built wheel for CPython 3.9 to /tmp/foo/bar/target/wheels/bar-0.1.0-cp39-cp39-linux_x86_64.whl
🛠️ Using zig for cross-compiling to x86_64-unknown-linux-gnu
/tmp/foo/bar/target/wheels/bar-0.1.0-cp39-cp39-linux_x86_64.whl
    Finished `release` profile [optimized] target(s) in 5.06s
📦 Built wheel for CPython 3.9 to /tmp/foo/baz/target/wheels/baz-0.1.0-cp39-cp39-linux_x86_64.whl
🛠️ Using zig for cross-compiling to x86_64-unknown-linux-gnu
/tmp/foo/baz/target/wheels/baz-0.1.0-cp39-cp39-linux_x86_64.whl
    Finished `release` profile [optimized] target(s) in 5.08s
📦 Built wheel for CPython 3.9 to /tmp/foo/target/wheels/foo-0.1.0-cp39-cp39-linux_x86_64.whl
🛠️ Using zig for cross-compiling to x86_64-unknown-linux-gnu
/tmp/foo/target/wheels/foo-0.1.0-cp39-cp39-linux_x86_64.whl
Successfully built build/bar-0.1.0-cp39-cp39-linux_x86_64.whl
Successfully built build/baz-0.1.0-cp39-cp39-linux_x86_64.whl
Successfully built build/foo-0.1.0-cp39-cp39-linux_x86_64.whl
iter: 1
# ...
iter: 2
[bar] Building wheel...
[baz] Building wheel...
[foo] Building wheel...
Running `maturin pep517 build-wheel -i /root/.cache/uv/builds-v0/.tmpPNNGqa/bin/python --compatibility off --zig`
Running `maturin pep517 build-wheel -i /root/.cache/uv/builds-v0/.tmpBRxODI/bin/python --compatibility off --zig`
Running `maturin pep517 build-wheel -i /root/.cache/uv/builds-v0/.tmpiYyaC3/bin/python --compatibility off --zig`
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
🔗 Found pyo3 bindings
🔗 Found pyo3 bindings
🔗 Found pyo3 bindings
🐍 Found CPython 3.9 at /root/.cache/uv/builds-v0/.tmpBRxODI/bin/python
📡 Using build options features from pyproject.toml
🐍 Found CPython 3.9 at /root/.cache/uv/builds-v0/.tmpiYyaC3/bin/python
📡 Using build options features from pyproject.toml
🐍 Found CPython 3.9 at /root/.cache/uv/builds-v0/.tmpPNNGqa/bin/python
📡 Using build options features from pyproject.toml
   Compiling pyo3-build-config v0.23.4
   Compiling pyo3-build-config v0.23.4
   Compiling pyo3-build-config v0.23.4
   Compiling pyo3-ffi v0.23.4
   Compiling pyo3-macros-backend v0.23.4
   Compiling pyo3 v0.23.4
   Compiling pyo3-ffi v0.23.4
   Compiling pyo3-macros-backend v0.23.4
   Compiling pyo3 v0.23.4
   Compiling pyo3-macros-backend v0.23.4
   Compiling pyo3-ffi v0.23.4
   Compiling pyo3 v0.23.4
   Compiling pyo3-macros v0.23.4
   Compiling pyo3-macros v0.23.4
   Compiling pyo3-macros v0.23.4
   Compiling baz v0.1.0 (/tmp/foo/baz)
   Compiling bar v0.1.0 (/tmp/foo/bar)
    Finished `release` profile [optimized] target(s) in 3.55s
   Compiling foo v0.1.0 (/tmp/foo)
📦 Built wheel for CPython 3.9 to /tmp/foo/baz/target/wheels/baz-0.1.0-cp39-cp39-linux_x86_64.whl
🛠️ Using zig for cross-compiling to x86_64-unknown-linux-gnu
/tmp/foo/baz/target/wheels/baz-0.1.0-cp39-cp39-linux_x86_64.whl
    Finished `release` profile [optimized] target(s) in 3.58s
📦 Built wheel for CPython 3.9 to /tmp/foo/bar/target/wheels/bar-0.1.0-cp39-cp39-linux_x86_64.whl
🛠️ Using zig for cross-compiling to x86_64-unknown-linux-gnu
/tmp/foo/bar/target/wheels/bar-0.1.0-cp39-cp39-linux_x86_64.whl
error: linking with `/root/.cache/cargo-zigbuild/0.19.7/zigcc-x86_64-unknown-linux-gnu-7d27.sh` failed: exit status: 127
  |
  = note: LC_ALL="C" PATH="/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/root/.cache/uv/builds-v0/.tmpBRxODI/bin:/root/.local/bin:/opt/zig:/root/.cargo/bin:/opt/llvm/18.1.8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" VSLANG="1033" "/root/.cache/cargo-zigbuild/0.19.7/zigcc-x86_64-unknown-linux-gnu-7d27.sh" "-Wl,--version-script=/tmp/rustcydd41p/list" "-Wl,--no-undefined-version" "-m64" "/tmp/rustcydd41p/symbols.o" "/tmp/foo/target/x86_64-unknown-linux-gnu/release/deps/foo.foo.6d4a9ca3f0096b57-cgu.0.rcgu.o" "/tmp/foo/target/x86_64-unknown-linux-gnu/release/deps/foo.foo.6d4a9ca3f0096b57-cgu.1.rcgu.o" "/tmp/foo/target/x86_64-unknown-linux-gnu/release/deps/foo.aiipswgzgnb0bin2q7a89u3t4.rcgu.o" "-Wl,--as-needed" "-Wl,-Bstatic" "/tmp/foo/target/x86_64-unknown-linux-gnu/release/deps/libpyo3-8afe8c16b2a858dc.rlib" "/tmp/foo/target/x86_64-unknown-linux-gnu/release/deps/libcfg_if-4b121bb8b89d189f.rlib" "/tmp/foo/target/x86_64-unknown-linux-gnu/release/deps/libmemoffset-2519831183e43da8.rlib" "/tmp/foo/target/x86_64-unknown-linux-gnu/release/deps/libonce_cell-cdaaef5e59b1d171.rlib" "/tmp/foo/target/x86_64-unknown-linux-gnu/release/deps/libpyo3_ffi-b9c081c714133a1e.rlib" "/tmp/foo/target/x86_64-unknown-linux-gnu/release/deps/liblibc-11df205d8be5b1be.rlib" "/tmp/foo/target/x86_64-unknown-linux-gnu/release/deps/libunindent-ae2b033a8dbb0775.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-6f060101dda10b7a.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-70f71d9a6d284c89.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-3dc783e4d6e04b19.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-4fffc92fe380873a.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-c572ac12fe8cba29.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-fc5479163f049547.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-abc6fce37da6e535.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-454117f0927a89af.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-302800adacdaa375.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-0b365bbc25bb52b1.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-ee3a5cb2982a2a9f.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-521236c0a42ef1a3.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-541d85b154f80ded.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-380aa0852ecf2621.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-099427f186ab3f66.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-1c29bac7d035f466.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-d193bce3b6600c65.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-75c6c8b4d83dd0d3.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-ad7808ee138565ae.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/tmp/foo/target/x86_64-unknown-linux-gnu/release/deps/libfoo.so" "-Wl,--gc-sections" "-shared" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-Wl,--strip-debug" "-nodefaultlibs"
  = note: /root/.cache/cargo-zigbuild/0.19.7/zigcc-x86_64-unknown-linux-gnu-7d27.sh: 2: exec: /root/.cache/uv/builds-v0/.tmpPNNGqa/bin/maturin: not found


error: could not compile `foo` (lib) due to 1 previous error
💥 maturin failed
  Caused by: Failed to build a native library through cargo
  Caused by: Cargo build finished with "exit status: 101": `env -u CARGO AR_x86_64_unknown_linux_gnu="/root/.cache/cargo-zigbuild/0.19.7/ar" CARGO_TARGET_APPLIES_TO_HOST="false" CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER="/root/.cache/cargo-zigbuild/0.19.7/zigcc-x86_64-unknown-linux-gnu-7d27.sh" CARGO_UNSTABLE_TARGET_APPLIES_TO_HOST="true" CARGO_ZIGBUILD_RUSTC_VERSION="1.84.1" CC_x86_64_unknown_linux_gnu="/root/.cache/cargo-zigbuild/0.19.7/zigcc-x86_64-unknown-linux-gnu-7d27.sh" CMAKE_TOOLCHAIN_FILE_x86_64_unknown_linux_gnu="/root/.cache/cargo-zigbuild/0.19.7/cmake/x86_64-unknown-linux-gnu-toolchain.cmake" CXX_x86_64_unknown_linux_gnu="/root/.cache/cargo-zigbuild/0.19.7/zigcxx-x86_64-unknown-linux-gnu-7d27.sh" PYO3_ENVIRONMENT_SIGNATURE="cpython-3.9-64bit" PYO3_PYTHON="/root/.cache/uv/builds-v0/.tmpBRxODI/bin/python" PYTHON_SYS_EXECUTABLE="/root/.cache/uv/builds-v0/.tmpBRxODI/bin/python" RANLIB_x86_64_unknown_linux_gnu="/root/.cache/cargo-zigbuild/0.19.7/zigranlib.sh" ZIG_COMMAND="/usr/local/bin/zig" __CARGO_TEST_CHANNEL_OVERRIDE_DO_NOT_USE_THIS="nightly" "cargo" "rustc" "--features" "pyo3/extension-module" "--target" "x86_64-unknown-linux-gnu" "--message-format" "json-render-diagnostics" "--manifest-path" "/tmp/foo/Cargo.toml" "--release" "--lib"`
🛠️ Using zig for cross-compiling to x86_64-unknown-linux-gnu
Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/root/.cache/uv/builds-v0/.tmpBRxODI/bin/python', '--compatibility', 'off', '--zig'] returned non-zero exit status 1
Successfully built build/bar-0.1.0-cp39-cp39-linux_x86_64.whl
Successfully built build/baz-0.1.0-cp39-cp39-linux_x86_64.whl
  x Failed to build `foo @ /tmp/foo`
  |-> The build backend returned an error
  `-> Call to `maturin.build_wheel` failed (exit status: 1)
      hint: This usually indicates a problem with the package or the build environment.

What happens is as follows:

cargo-zigbuild/src/zig.rs

Line 1437 in 1ddfd9b

"exec \"{}\" zig {} -- {} \"$@\"",

When the build is driven by maturin instead of cargo, the generated linker script points to maturin inside a temporary build venv. Each build venv is destroyed by uv immediately after the build finishes. Thus, if multiple builds ran in parallel, each with their own build venv and maturin binary, but pointing to the the same linker script in the common cache directory, it's not guaranteed that the linker script script still points to a valid maturin binary when it's called later.

I think the easiest fix may be to include env::current_exe() in the hash of the linker script name.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions