Rust 1.85.0 Breaks Build Process

[cbgbt]

Summary
Bottlerocket fails to build if the default toolchain on your build host is Rust 1.85.0. We are investigating. In the meantime, you can workaround the issue by locking your Rust toolchain to 1.84.1:

rustup toolchain default 1.84.1

---

Platform I'm building on:
Amazon Linux 2023, aarch64

What I expected to happen:
The following should succeed:

rustup toolchain install 1.85.0
cargo +1.85.0 make -e BUILDSYS_ARCH=aarch64 -e BUILDSYS_VARIANT=aws-k8s-1.30 check-licenses

What actually happened:

The command errors with the following:

[cargo-make][1] INFO - Running Task: check-licenses
2025-02-26 20:12:05 [ERROR] failed to fetch crates:     Updating crates.io index
    Updating git repository `https://github.com/bottlerocket-os/bottlerocket-settings-sdk`
warning: spurious network error (3 tries remaining): failed to resolve address for github.com: Temporary failure in name resolution; class=Net (12)
warning: spurious network error (2 tries remaining): failed to resolve address for github.com: Temporary failure in name resolution; class=Net (12)
warning: spurious network error (1 tries remaining): failed to resolve address for github.com: Temporary failure in name resolution; class=Net (12)
error: failed to get `bottlerocket-settings-models` as a dependency of package `models v0.1.0 (/tmp/sources/models)`

Caused by:
  failed to load source for dependency `bottlerocket-settings-models`

Caused by:
  Unable to update https://github.com/bottlerocket-os/bottlerocket-settings-sdk?tag=bottlerocket-settings-models-v0.7.0#2a7c0986

Caused by:
  failed to fetch into: /tmp/.cargo/git/db/bottlerocket-settings-sdk-3ffd24dec00e96c2

Caused by:
  network failure seems to have happened
  if a proxy or similar is necessary `net.git-fetch-with-cli` may help here
  https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli

Caused by:
  failed to resolve address for github.com: Temporary failure in name resolution; class=Net (12)

2025-02-26 20:12:05 [ERROR] `cargo metadata` exited with an error:     Updating crates.io index
    Updating git repository `https://github.com/bottlerocket-os/bottlerocket-settings-sdk`
warning: spurious network error (3 tries remaining): failed to resolve address for github.com: Temporary failure in name resolution; class=Net (12)
warning: spurious network error (2 tries remaining): failed to resolve address for github.com: Temporary failure in name resolution; class=Net (12)
warning: spurious network error (1 tries remaining): failed to resolve address for github.com: Temporary failure in name resolution; class=Net (12)
error: failed to get `bottlerocket-settings-models` as a dependency of package `models v0.1.0 (/tmp/sources/models)`

Caused by:
  failed to load source for dependency `bottlerocket-settings-models`

Caused by:
  Unable to update https://github.com/bottlerocket-os/bottlerocket-settings-sdk?tag=bottlerocket-settings-models-v0.7.0#2a7c0986

Caused by:
  failed to fetch into: /tmp/.cargo/git/db/bottlerocket-settings-sdk-3ffd24dec00e96c2

[cbgbt]

This appears to be consistent across platforms if we're on the latest release of Rust, which was released on 2/20/2025 and contained large changes for the newest Rust edition.

Presumably one of these changes was to the way that git-based crate metadata is stored in the .cargo cache. Possibly the issue here occurs because of the mismatch of Rust version on the build host and the one used within the SDK.

[cbgbt]

I took a peek at .cargo/git/ and see the following:

Using Rust/Cargo 1.85.0:

.cargo/git
├── CACHEDIR.TAG
├── checkouts
│   └── bottlerocket-settings-sdk-0fcecbd216d3a870
│       ├── 2a7c098

Using Rust/Cargo 1.84.0:

.cargo/git
├── CACHEDIR.TAG
├── checkouts
│   └── bottlerocket-settings-sdk-3ffd24dec00e96c2
│       ├── 2a7c098

The cargo changelog for 1.85.0 says the following:

❗️ Cargo now uses a cross-platform hash algorithm from rustc-stable-hash. As a result, the hash part of paths to dependency caches (e.g., $CARGO_HOME/registry/index/index.crates.io-) will change. This will trigger re-downloads of registry indices and .crate tarballs, as well as re-cloning of Git dependencies. #14917

Wanting to determine if this is due to using git dependencies or affects all crates, I took a look at .cargo/registry:

Rust/Cargo 1.85.0:

.cargo/registry
├── cache
│   └── index.crates.io-1949cf8c6b5b557f

Rust/Cargo 1.84.0:

.cargo/registry
├── cache
│   └── index.crates.io-6f17d22bba15001f

---

So indeed, it appears that the issue is unrelated to git dependencies and is likely due to passing off a Cargo cache from 1.85.0 from the host into the SDK, which is using 1.84.1. These versions use a different hashing algorithm to find cached Rust sources.

[cbgbt]

We can possibly fix this by always vendoring Rust crates using the SDK (rather than the host's Rust toolchain), but with host networking enabled in the container.

[jmt-lab]

After investigating it turns out the core of the problem is that we are doing a manual hack to vendor cargo sources. We are not actually using cargo vendor, instead we are copying a resolved local .cargo/registry path which is always specific and tied to cargo versions, we should use cargo vendor instead as it won't have this version collision

[jmt-lab]

This is fixed.