Skip to content

cargo_build_script is non determnistic when compiling cc_libraries #3481

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

cargo_build_script is non determnistic when compiling cc_libraries #3481

wants to merge 2 commits into from

Conversation

sebas-zip
Copy link

run:
cd examples/ring_example
./test_deterministic.sh

=== Testing Deterministic Build for Ring Crate ===
Cleaning up previous test store...
--- First Build ---
INFO: Analyzed target @@rules_rust++crate+crates//:ring (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target @@rules_rust++crate+crates__ring-0.17.14//:ring up-to-date:
  bazel-bin/external/rules_rust++crate+crates__ring-0.17.14/libring-2707094885.rlib
INFO: Elapsed time: 0.143s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
Copying first build artifacts...
First build artifacts stored in: ./deterministic_test_store/build1
total 15212
drwxrwxr-x 2 sesovero sesovero    4096 Jun 23 22:59 .
drwxrwxr-x 4 sesovero sesovero    4096 Jun 23 22:59 ..
-r-xr-xr-x 1 sesovero sesovero 6605936 Jun 23 22:59 libring-2707094885.rlib
-r-xr-xr-x 1 sesovero sesovero 2257450 Jun 23 22:59 libring_core_0_17_14_.a
-r-xr-xr-x 1 sesovero sesovero   90408 Jun 23 22:59 libring_core_0_17_14__test.a
-r-xr-xr-x 1 sesovero sesovero 6605936 Jun 23 22:59 ring_main_output
--- Cleaning Build ---
INFO: Starting clean (this may take a while). Use --async if the clean takes more than several minutes.
--- Second Build ---
INFO: Analyzed target @@rules_rust++crate+crates//:ring (158 packages loaded, 4055 targets configured).
INFO: Found 1 target...
Target @@rules_rust++crate+crates__ring-0.17.14//:ring up-to-date:
  bazel-bin/external/rules_rust++crate+crates__ring-0.17.14/libring-2707094885.rlib
INFO: Elapsed time: 40.829s, Critical Path: 39.88s
INFO: 233 processes: 138 internal, 95 linux-sandbox.
INFO: Build completed successfully, 233 total actions
Copying second build artifacts...
Second build artifacts stored in: ./deterministic_test_store/build2
total 15212
drwxrwxr-x 2 sesovero sesovero    4096 Jun 23 22:59 .
drwxrwxr-x 4 sesovero sesovero    4096 Jun 23 22:59 ..
-r-xr-xr-x 1 sesovero sesovero 6605936 Jun 23 22:59 libring-2707094885.rlib
-r-xr-xr-x 1 sesovero sesovero 2257450 Jun 23 22:59 libring_core_0_17_14_.a
-r-xr-xr-x 1 sesovero sesovero   90408 Jun 23 22:59 libring_core_0_17_14__test.a
-r-xr-xr-x 1 sesovero sesovero 6605936 Jun 23 22:59 ring_main_output
--- Comparing Build Outputs ---
❌ FAILURE: Builds are NOT deterministic. Differences found:

Binary files ./deterministic_test_store/build1/libring-2707094885.rlib and ./deterministic_test_store/build2/libring-2707094885.rlib differ
Binary files ./deterministic_test_store/build1/libring_core_0_17_14_.a and ./deterministic_test_store/build2/libring_core_0_17_14_.a differ
Binary files ./deterministic_test_store/build1/libring_core_0_17_14__test.a and ./deterministic_test_store/build2/libring_core_0_17_14__test.a differ
Binary files ./deterministic_test_store/build1/ring_main_output and ./deterministic_test_store/build2/ring_main_output differ

File hashes comparison:
Build 1:
12de80831074533a8fa80214433c587fd2f8847b9fb1b6346e200ae1ab504007  ./deterministic_test_store/build1/ring_main_output
33908167cbdc9c0eeeb8ddb3b08e8fb948b74991dc92707717da5c54b7f7be47  ./deterministic_test_store/build1/libring_core_0_17_14__test.a
12de80831074533a8fa80214433c587fd2f8847b9fb1b6346e200ae1ab504007  ./deterministic_test_store/build1/libring-2707094885.rlib
19cd5d498114b7c9040ebcfffe083239d4a89c933ea2d6557e10e6b51582ca2c  ./deterministic_test_store/build1/libring_core_0_17_14_.a

Build 2:
5a06967c7bdc54019cd4a9c8432802ffd9da6ada87f2aff27902b34c2ac85e1b  ./deterministic_test_store/build2/ring_main_output
2d530a234ea73f9952253a80f266a9c0a4f9276e6dc60eda7ef47a802520449f  ./deterministic_test_store/build2/libring_core_0_17_14__test.a
5a06967c7bdc54019cd4a9c8432802ffd9da6ada87f2aff27902b34c2ac85e1b  ./deterministic_test_store/build2/libring-2707094885.rlib
070b95e6218aa31978da520e2a908c4cda2f75d1269b756dae590c2530edd17d  ./deterministic_test_store/build2/libring_core_0_17_14_.a

=== Test Complete ===
Build artifacts stored in: ./deterministic_test_store

@sebas-zip sebas-zip changed the title cargo_build_script is non determnistic when compile cc_libraries cargo_build_script is non determnistic when compiling cc_libraries Jun 23, 2025
@sebas-zip sebas-zip force-pushed the sebas/ring_example branch from 46e0423 to e5e18d2 Compare June 25, 2025 22:13
@sebas-zip
Copy link
Author

right now this example demonstrates that DEBUG in CC_libraries built with build.rs is non deterministic.

If you want to see the LTO symbols be non deterministic I think you have to setup a cross compilation which I haven't setup yet.

@illicitonion
Copy link
Collaborator

Thanks for putting this together!

I don't quite see the same thing when I run it - I see:

❌ File missing in second build: libcfg_if-1498835963.rlib
❌ File missing in second build: libgetrandom-2908067148.rlib
❌ File missing in second build: liblibc-2535035137.rlib
✅ Identical: libring-2707094885.rlib
❌ File missing in second build: libuntrusted-798176677.rlib

❌ FAILURE: Builds are NOT deterministic.

I've tried this on both macOS and Ubuntu 20.04 - what platform are you running on? Maybe you have a docker image you can point me at? And possibly share the diffoscope output?

@sebas-zip
Copy link
Author

sebas-zip commented Jul 1, 2025 •
edited
Loading

shucks... it works on ubuntu 22.04 for me and osx 15.5

lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.5 LTS
Release:	22.04
Codename:	jammy
sesovero@ip-10-50-21-157:~/repos/rules_rust/examples/ring_example$ ./test_deterministic.sh
=== Testing Deterministic Build for Ring Crate (.rlib files) ===
Target: Native
Building for native target
Cleaning up previous test store...
--- First Build ---
INFO: Analyzed target @@rules_rust++crate+crates//:ring (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target @@rules_rust++crate+crates__ring-0.17.14//:ring up-to-date:
  bazel-bin/external/rules_rust++crate+crates__ring-0.17.14/libring-2707094885.rlib
INFO: Elapsed time: 0.411s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
Copying first build .rlib artifacts...
First build .rlib artifacts stored in: ./deterministic_test_store_native/build1
total 11048
drwxrwxr-x 2 sesovero sesovero    4096 Jul  1 21:00 .
drwxrwxr-x 4 sesovero sesovero    4096 Jul  1 21:00 ..
-r-xr-xr-x 1 sesovero sesovero 1428100 Jul  1 21:00 libcc-1847392201.rlib
-r-xr-xr-x 1 sesovero sesovero    8284 Jul  1 21:00 libcfg_if-1498835963.rlib
-r-xr-xr-x 1 sesovero sesovero  164110 Jul  1 21:00 libgetrandom-2908067148.rlib
-r-xr-xr-x 1 sesovero sesovero 3588426 Jul  1 21:00 liblibc-2535035137.rlib
-r-xr-xr-x 1 sesovero sesovero 5967818 Jul  1 21:00 libring-2707094885.rlib
-r-xr-xr-x 1 sesovero sesovero   82994 Jul  1 21:00 libshlex-1192625423.rlib
-r-xr-xr-x 1 sesovero sesovero   45436 Jul  1 21:00 libuntrusted-798176677.rlib
--- Cleaning Build ---
INFO: Starting clean (this may take a while). Use --async if the clean takes more than several minutes.
--- Second Build ---
INFO: Analyzed target @@rules_rust++crate+crates//:ring (158 packages loaded, 3996 targets configured).
INFO: Found 1 target...
Target @@rules_rust++crate+crates__ring-0.17.14//:ring up-to-date:
  bazel-bin/external/rules_rust++crate+crates__ring-0.17.14/libring-2707094885.rlib
INFO: Elapsed time: 49.502s, Critical Path: 47.83s
INFO: 235 processes: 140 internal, 95 linux-sandbox.
INFO: Build completed successfully, 235 total actions
Copying second build .rlib artifacts...
Second build .rlib artifacts stored in: ./deterministic_test_store_native/build2
total 11048
drwxrwxr-x 2 sesovero sesovero    4096 Jul  1 21:01 .
drwxrwxr-x 4 sesovero sesovero    4096 Jul  1 21:00 ..
-r-xr-xr-x 1 sesovero sesovero 1428100 Jul  1 21:01 libcc-1847392201.rlib
-r-xr-xr-x 1 sesovero sesovero    8284 Jul  1 21:01 libcfg_if-1498835963.rlib
-r-xr-xr-x 1 sesovero sesovero  164110 Jul  1 21:01 libgetrandom-2908067148.rlib
-r-xr-xr-x 1 sesovero sesovero 3588426 Jul  1 21:01 liblibc-2535035137.rlib
-r-xr-xr-x 1 sesovero sesovero 5967818 Jul  1 21:01 libring-2707094885.rlib
-r-xr-xr-x 1 sesovero sesovero   82994 Jul  1 21:01 libshlex-1192625423.rlib
-r-xr-xr-x 1 sesovero sesovero   45436 Jul  1 21:01 libuntrusted-798176677.rlib
--- Comparing Build Outputs ---
Comparing .rlib files individually...
✅ Identical: libcc-1847392201.rlib
✅ Identical: libcfg_if-1498835963.rlib
✅ Identical: libgetrandom-2908067148.rlib
✅ Identical: liblibc-2535035137.rlib
❌ DIFFERENCE FOUND: libring-2707094885.rlib
  Hash 1: 0d5def52e74a57beaf653e9bde5cd71477631477acace2109ccc2cd03c080564
  Hash 2: b209fdf19a370cd76f26a791aa4d38af2f078e8309daea8cde6df8a727dc0436
✅ Identical: libshlex-1192625423.rlib
✅ Identical: libuntrusted-798176677.rlib

❌ FAILURE: Builds are NOT deterministic.
Different .rlib files: 1 out of 7

Debug commands for different files:
  For libring-2707094885.rlib:
    diffoscope './deterministic_test_store_native/build1/libring-2707094885.rlib' './deterministic_test_store_native/build2/libring-2707094885.rlib'
    Or upload to https://try.diffoscope.org/

=== Test Complete ===
.rlib artifacts stored in: ./deterministic_test_store_native

@sebas-zip
Copy link
Author

@illicitonion I see debug symbols changing in the ring library. I think to get the lto symbols to change it has to be setup in cross compilation mode
ring_diff.txt.zip

sebas-zip
sebas-zip commented Jul 3, 2025
View reviewed changes
examples/ring_example/test_deterministic.sh

# Define the build command with flags (modify this to add/change flags)
# Example: BUILD_CMD="bazel build --noremote_accept_cached --verbose_failures --sandbox_debug"
BUILD_CMD="bazel build --noremote_accept_cached --@rules_rust//rust/settings:lto=thin"
Copy link
Author

@sebas-zip sebas-zip Jul 3, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should build the target for you locally if you don't want to use the shell script:
cd examples/ring_example
bazel build --noremote_accept_cached --@rules_rust//rust/settings:lto=thin @crates//:ring

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sebas-zip @illicitonion