Skip to content

Sporadic failures with parallel builds through maturin #318

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
tuxu opened this issue Feb 12, 2025 · 1 comment · May be fixed by #319
Open

Sporadic failures with parallel builds through maturin #318

tuxu opened this issue Feb 12, 2025 · 1 comment · May be fixed by #319

Comments

@tuxu
Copy link

tuxu commented Feb 12, 2025

I ran into a race condition with cargo-zigbuild's generated linker script when multiple such builds are driven in parallel through maturin and uv. I initially blamed uv, because the error message indicated an issue with the lifetime of the temporary build venv, however @konstin pointed out that the linker script generation here is not safe (astral-sh/uv#11444 (comment)).

Steps to reproduce: 
# Running inside a ubuntu:22.04 container with uv / maturin / zig
# CPU is Intel i7-13700K
root@23be7dfc1db1:/tmp/foo# uv --version
uv 0.5.29
root@23be7dfc1db1:/tmp/foo# maturin --version
maturin 1.8.2
root@23be7dfc1db1:/tmp# maturin new foo -b pyo3
  ✨ Done! New project created foo
root@23be7dfc1db1:/tmp# cd foo
root@23be7dfc1db1:/tmp/foo# maturin new bar -b pyo3
  ✨ Done! New project created bar
root@23be7dfc1db1:/tmp/foo# maturin new baz -b pyo3
  ✨ Done! New project created baz
root@23be7dfc1db1:/tmp/foo# echo -e "[tool.uv.workspace]\nmembers = [\"bar\", \"baz\"]\n" >> pyproject.toml
root@23be7dfc1db1:/tmp/foo# iter=0; while uv build --wheel --all-packages --out-dir build --config-setting "build-args=--zig" ; do ((iter++)); echo "iter: $iter" ; done
[bar] Building wheel...
[baz] Building wheel...
[foo] Building wheel...
Running `maturin pep517 build-wheel -i /root/.cache/uv/builds-v0/.tmpNCK6Wi/bin/python --compatibility off --zig`
Running `maturin pep517 build-wheel -i /root/.cache/uv/builds-v0/.tmpVvuph8/bin/python --compatibility off --zig`
    Updating crates.io index
    Blocking waiting for file lock on package cache
Running `maturin pep517 build-wheel -i /root/.cache/uv/builds-v0/.tmp3joi2Z/bin/python --compatibility off --zig`
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Updating crates.io index
    Blocking waiting for file lock on package cache
    Updating crates.io index
    Blocking waiting for file lock on package cache
     Locking 19 packages to latest compatible versions
    Blocking waiting for file lock on package cache
     Locking 19 packages to latest compatible versions
    Blocking waiting for file lock on package cache
     Locking 19 packages to latest compatible versions
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
🔗 Found pyo3 bindings
🔗 Found pyo3 bindings
🔗 Found pyo3 bindings
🐍 Found CPython 3.9 at /root/.cache/uv/builds-v0/.tmpNCK6Wi/bin/python
📡 Using build options features from pyproject.toml
🐍 Found CPython 3.9 at /root/.cache/uv/builds-v0/.tmp3joi2Z/bin/python
📡 Using build options features from pyproject.toml
🐍 Found CPython 3.9 at /root/.cache/uv/builds-v0/.tmpVvuph8/bin/python
📡 Using build options features from pyproject.toml
    Blocking waiting for file lock on package cache
   Compiling target-lexicon v0.12.16
   Compiling once_cell v1.20.3
   Compiling proc-macro2 v1.0.93
   Compiling unicode-ident v1.0.16
   Compiling libc v0.2.169
   Compiling autocfg v1.4.0
   Compiling heck v0.5.0
   Compiling cfg-if v1.0.0
   Compiling indoc v2.0.5
   Compiling unindent v0.2.3
    Blocking waiting for file lock on package cache
   Compiling target-lexicon v0.12.16
   Compiling once_cell v1.20.3
   Compiling proc-macro2 v1.0.93
   Compiling unicode-ident v1.0.16
   Compiling autocfg v1.4.0
   Compiling libc v0.2.169
   Compiling heck v0.5.0
   Compiling target-lexicon v0.12.16
   Compiling cfg-if v1.0.0
   Compiling once_cell v1.20.3
   Compiling proc-macro2 v1.0.93
   Compiling unicode-ident v1.0.16
   Compiling libc v0.2.169
   Compiling autocfg v1.4.0
   Compiling indoc v2.0.5
   Compiling heck v0.5.0
   Compiling unindent v0.2.3
   Compiling indoc v2.0.5
   Compiling unindent v0.2.3
   Compiling cfg-if v1.0.0
   Compiling memoffset v0.9.1
   Compiling memoffset v0.9.1
   Compiling memoffset v0.9.1
   Compiling pyo3-build-config v0.23.4
   Compiling pyo3-build-config v0.23.4
   Compiling quote v1.0.38
   Compiling pyo3-build-config v0.23.4
   Compiling quote v1.0.38
   Compiling quote v1.0.38
   Compiling syn v2.0.98
   Compiling syn v2.0.98
   Compiling syn v2.0.98
   Compiling pyo3-macros-backend v0.23.4
   Compiling pyo3-ffi v0.23.4
   Compiling pyo3 v0.23.4
   Compiling pyo3-ffi v0.23.4
   Compiling pyo3-macros-backend v0.23.4
   Compiling pyo3 v0.23.4
   Compiling pyo3-macros-backend v0.23.4
   Compiling pyo3-ffi v0.23.4
   Compiling pyo3 v0.23.4
   Compiling pyo3-macros v0.23.4
   Compiling pyo3-macros v0.23.4
   Compiling pyo3-macros v0.23.4
   Compiling bar v0.1.0 (/tmp/foo/bar)
   Compiling baz v0.1.0 (/tmp/foo/baz)
   Compiling foo v0.1.0 (/tmp/foo)
    Finished `release` profile [optimized] target(s) in 4.99s
📦 Built wheel for CPython 3.9 to /tmp/foo/bar/target/wheels/bar-0.1.0-cp39-cp39-linux_x86_64.whl
🛠️ Using zig for cross-compiling to x86_64-unknown-linux-gnu
/tmp/foo/bar/target/wheels/bar-0.1.0-cp39-cp39-linux_x86_64.whl
    Finished `release` profile [optimized] target(s) in 5.06s
📦 Built wheel for CPython 3.9 to /tmp/foo/baz/target/wheels/baz-0.1.0-cp39-cp39-linux_x86_64.whl
🛠️ Using zig for cross-compiling to x86_64-unknown-linux-gnu
/tmp/foo/baz/target/wheels/baz-0.1.0-cp39-cp39-linux_x86_64.whl
    Finished `release` profile [optimized] target(s) in 5.08s
📦 Built wheel for CPython 3.9 to /tmp/foo/target/wheels/foo-0.1.0-cp39-cp39-linux_x86_64.whl
🛠️ Using zig for cross-compiling to x86_64-unknown-linux-gnu
/tmp/foo/target/wheels/foo-0.1.0-cp39-cp39-linux_x86_64.whl
Successfully built build/bar-0.1.0-cp39-cp39-linux_x86_64.whl
Successfully built build/baz-0.1.0-cp39-cp39-linux_x86_64.whl
Successfully built build/foo-0.1.0-cp39-cp39-linux_x86_64.whl
iter: 1
# ...
iter: 2
[bar] Building wheel...
[baz] Building wheel...
[foo] Building wheel...
Running `maturin pep517 build-wheel -i /root/.cache/uv/builds-v0/.tmpPNNGqa/bin/python --compatibility off --zig`
Running `maturin pep517 build-wheel -i /root/.cache/uv/builds-v0/.tmpBRxODI/bin/python --compatibility off --zig`
Running `maturin pep517 build-wheel -i /root/.cache/uv/builds-v0/.tmpiYyaC3/bin/python --compatibility off --zig`
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
🔗 Found pyo3 bindings
🔗 Found pyo3 bindings
🔗 Found pyo3 bindings
🐍 Found CPython 3.9 at /root/.cache/uv/builds-v0/.tmpBRxODI/bin/python
📡 Using build options features from pyproject.toml
🐍 Found CPython 3.9 at /root/.cache/uv/builds-v0/.tmpiYyaC3/bin/python
📡 Using build options features from pyproject.toml
🐍 Found CPython 3.9 at /root/.cache/uv/builds-v0/.tmpPNNGqa/bin/python
📡 Using build options features from pyproject.toml
   Compiling pyo3-build-config v0.23.4
   Compiling pyo3-build-config v0.23.4
   Compiling pyo3-build-config v0.23.4
   Compiling pyo3-ffi v0.23.4
   Compiling pyo3-macros-backend v0.23.4
   Compiling pyo3 v0.23.4
   Compiling pyo3-ffi v0.23.4
   Compiling pyo3-macros-backend v0.23.4
   Compiling pyo3 v0.23.4
   Compiling pyo3-macros-backend v0.23.4
   Compiling pyo3-ffi v0.23.4
   Compiling pyo3 v0.23.4
   Compiling pyo3-macros v0.23.4
   Compiling pyo3-macros v0.23.4
   Compiling pyo3-macros v0.23.4
   Compiling baz v0.1.0 (/tmp/foo/baz)
   Compiling bar v0.1.0 (/tmp/foo/bar)
    Finished `release` profile [optimized] target(s) in 3.55s
   Compiling foo v0.1.0 (/tmp/foo)
📦 Built wheel for CPython 3.9 to /tmp/foo/baz/target/wheels/baz-0.1.0-cp39-cp39-linux_x86_64.whl
🛠️ Using zig for cross-compiling to x86_64-unknown-linux-gnu
/tmp/foo/baz/target/wheels/baz-0.1.0-cp39-cp39-linux_x86_64.whl
    Finished `release` profile [optimized] target(s) in 3.58s
📦 Built wheel for CPython 3.9 to /tmp/foo/bar/target/wheels/bar-0.1.0-cp39-cp39-linux_x86_64.whl
🛠️ Using zig for cross-compiling to x86_64-unknown-linux-gnu
/tmp/foo/bar/target/wheels/bar-0.1.0-cp39-cp39-linux_x86_64.whl
error: linking with `/root/.cache/cargo-zigbuild/0.19.7/zigcc-x86_64-unknown-linux-gnu-7d27.sh` failed: exit status: 127
  |
  = note: LC_ALL="C" PATH="/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/root/.cache/uv/builds-v0/.tmpBRxODI/bin:/root/.local/bin:/opt/zig:/root/.cargo/bin:/opt/llvm/18.1.8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" VSLANG="1033" "/root/.cache/cargo-zigbuild/0.19.7/zigcc-x86_64-unknown-linux-gnu-7d27.sh" "-Wl,--version-script=/tmp/rustcydd41p/list" "-Wl,--no-undefined-version" "-m64" "/tmp/rustcydd41p/symbols.o" "/tmp/foo/target/x86_64-unknown-linux-gnu/release/deps/foo.foo.6d4a9ca3f0096b57-cgu.0.rcgu.o" "/tmp/foo/target/x86_64-unknown-linux-gnu/release/deps/foo.foo.6d4a9ca3f0096b57-cgu.1.rcgu.o" "/tmp/foo/target/x86_64-unknown-linux-gnu/release/deps/foo.aiipswgzgnb0bin2q7a89u3t4.rcgu.o" "-Wl,--as-needed" "-Wl,-Bstatic" "/tmp/foo/target/x86_64-unknown-linux-gnu/release/deps/libpyo3-8afe8c16b2a858dc.rlib" "/tmp/foo/target/x86_64-unknown-linux-gnu/release/deps/libcfg_if-4b121bb8b89d189f.rlib" "/tmp/foo/target/x86_64-unknown-linux-gnu/release/deps/libmemoffset-2519831183e43da8.rlib" "/tmp/foo/target/x86_64-unknown-linux-gnu/release/deps/libonce_cell-cdaaef5e59b1d171.rlib" "/tmp/foo/target/x86_64-unknown-linux-gnu/release/deps/libpyo3_ffi-b9c081c714133a1e.rlib" "/tmp/foo/target/x86_64-unknown-linux-gnu/release/deps/liblibc-11df205d8be5b1be.rlib" "/tmp/foo/target/x86_64-unknown-linux-gnu/release/deps/libunindent-ae2b033a8dbb0775.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-6f060101dda10b7a.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-70f71d9a6d284c89.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-3dc783e4d6e04b19.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-4fffc92fe380873a.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-c572ac12fe8cba29.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-fc5479163f049547.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-abc6fce37da6e535.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-454117f0927a89af.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-302800adacdaa375.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-0b365bbc25bb52b1.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-ee3a5cb2982a2a9f.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-521236c0a42ef1a3.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-541d85b154f80ded.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-380aa0852ecf2621.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-099427f186ab3f66.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-1c29bac7d035f466.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-d193bce3b6600c65.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-75c6c8b4d83dd0d3.rlib" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-ad7808ee138565ae.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/root/.rustup/toolchains/1.84.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/tmp/foo/target/x86_64-unknown-linux-gnu/release/deps/libfoo.so" "-Wl,--gc-sections" "-shared" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-Wl,--strip-debug" "-nodefaultlibs"
  = note: /root/.cache/cargo-zigbuild/0.19.7/zigcc-x86_64-unknown-linux-gnu-7d27.sh: 2: exec: /root/.cache/uv/builds-v0/.tmpPNNGqa/bin/maturin: not found


error: could not compile `foo` (lib) due to 1 previous error
💥 maturin failed
  Caused by: Failed to build a native library through cargo
  Caused by: Cargo build finished with "exit status: 101": `env -u CARGO AR_x86_64_unknown_linux_gnu="/root/.cache/cargo-zigbuild/0.19.7/ar" CARGO_TARGET_APPLIES_TO_HOST="false" CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER="/root/.cache/cargo-zigbuild/0.19.7/zigcc-x86_64-unknown-linux-gnu-7d27.sh" CARGO_UNSTABLE_TARGET_APPLIES_TO_HOST="true" CARGO_ZIGBUILD_RUSTC_VERSION="1.84.1" CC_x86_64_unknown_linux_gnu="/root/.cache/cargo-zigbuild/0.19.7/zigcc-x86_64-unknown-linux-gnu-7d27.sh" CMAKE_TOOLCHAIN_FILE_x86_64_unknown_linux_gnu="/root/.cache/cargo-zigbuild/0.19.7/cmake/x86_64-unknown-linux-gnu-toolchain.cmake" CXX_x86_64_unknown_linux_gnu="/root/.cache/cargo-zigbuild/0.19.7/zigcxx-x86_64-unknown-linux-gnu-7d27.sh" PYO3_ENVIRONMENT_SIGNATURE="cpython-3.9-64bit" PYO3_PYTHON="/root/.cache/uv/builds-v0/.tmpBRxODI/bin/python" PYTHON_SYS_EXECUTABLE="/root/.cache/uv/builds-v0/.tmpBRxODI/bin/python" RANLIB_x86_64_unknown_linux_gnu="/root/.cache/cargo-zigbuild/0.19.7/zigranlib.sh" ZIG_COMMAND="/usr/local/bin/zig" __CARGO_TEST_CHANNEL_OVERRIDE_DO_NOT_USE_THIS="nightly" "cargo" "rustc" "--features" "pyo3/extension-module" "--target" "x86_64-unknown-linux-gnu" "--message-format" "json-render-diagnostics" "--manifest-path" "/tmp/foo/Cargo.toml" "--release" "--lib"`
🛠️ Using zig for cross-compiling to x86_64-unknown-linux-gnu
Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/root/.cache/uv/builds-v0/.tmpBRxODI/bin/python', '--compatibility', 'off', '--zig'] returned non-zero exit status 1
Successfully built build/bar-0.1.0-cp39-cp39-linux_x86_64.whl
Successfully built build/baz-0.1.0-cp39-cp39-linux_x86_64.whl
  x Failed to build `foo @ /tmp/foo`
  |-> The build backend returned an error
  `-> Call to `maturin.build_wheel` failed (exit status: 1)
      hint: This usually indicates a problem with the package or the build environment.

What happens is as follows:

cargo-zigbuild/src/zig.rs

Line 1437 in 1ddfd9b

"exec \"{}\" zig {} -- {} \"$@\"",

When the build is driven by maturin instead of cargo, the generated linker script points to maturin inside a temporary build venv. Each build venv is destroyed by uv immediately after the build finishes. Thus, if multiple builds ran in parallel, each with their own build venv and maturin binary, but pointing to the the same linker script in the common cache directory, it's not guaranteed that the linker script script still points to a valid maturin binary when it's called later.

I think the easiest fix may be to include env::current_exe() in the hash of the linker script name.
tuxu added a commit to tuxu/cargo-zigbuild that referenced this issue Feb 13, 2025
@tuxu
Include current_exe in hash for linker wrapper path
fb95efa 
Ensures that unique linker wrapper scripts are generated even when
current_exe is different. Fixes an issue with concurrent builds driven
through maturin.

Closes rust-cross#318.
@tuxu tuxu linked a pull request Feb 13, 2025 that will close this issue
Open
tuxu added a commit to tuxu/cargo-zigbuild that referenced this issue Feb 13, 2025
@tuxu
Include current_exe in hash for linker wrapper path
a97f85d 
Ensures that unique linker wrapper scripts are generated even when
current_exe is different. Fixes an issue with concurrent builds driven
through maturin.

Closes rust-cross#318.
@tuxu
Copy link
Author

tuxu commented Feb 13, 2025

While testing, I noticed that there's another issue. The symlinks in the cache directory also point to maturin, and it's possible to run into this:

💥 maturin failed
  Caused by: Failed to build a native library through cargo
  Caused by: failed to remove file `/root/.cache/cargo-zigbuild/0.19.8/ar`: No such file or directory (os error 2)

tuxu added a commit to tuxu/cargo-zigbuild that referenced this issue Feb 13, 2025
@tuxu
Include current_exe in hash for linker wrapper path
b4e0ffb 
Ensures that unique linker wrapper scripts are generated even when
current_exe is different. Fixes an issue with concurrent builds driven
through maturin.

Closes rust-cross#318.
tuxu added a commit to tuxu/cargo-zigbuild that referenced this issue Feb 13, 2025
@tuxu
Include current_exe in hash for linker wrapper path
77a660b 
Ensures that unique linker wrapper scripts are generated even when
current_exe is different. Fixes an issue with concurrent builds driven
through maturin.

Closes rust-cross#318.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Include current_exe in hash for linker wrapper path tuxu/cargo-zigbuild
1 participant
@tuxu