Crash when using `per-package-target` and `build-std`

[Qix-]

Problem

Reproduction case: https://github.com/Qix-/cargo-crash-repro-2023-3-28

When using the per-package-target unstable feature along with build-std et al, cargo crashes with a panic.

Panic backtrace

thread 'main' panicked at 'no entry found for key', src/tools/cargo/src/cargo/core/compiler/unit_dependencies.rs:177:25
stack backtrace:
   0:     0x7f1b7558938a - std::backtrace_rs::backtrace::libunwind::trace::h33a8ebdd070a2849
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x7f1b7558938a - std::backtrace_rs::backtrace::trace_unsynchronized::hdcf243f187b50d8d
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x7f1b7558938a - std::sys_common::backtrace::_print_fmt::h447195462962d9fa
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/std/src/sys_common/backtrace.rs:65:5
   3:     0x7f1b7558938a - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h030527b6b1cd3ea0
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x7f1b755b8ddf - core::fmt::write::h820832a7aab88d9e
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/core/src/fmt/mod.rs:1254:17
   5:     0x7f1b75581d35 - std::io::Write::write_fmt::h8f101bdb654e5017
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/std/src/io/mod.rs:1698:15
   6:     0x7f1b75589155 - std::sys_common::backtrace::_print::h774dcd3ec539df8d
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/std/src/sys_common/backtrace.rs:47:5
   7:     0x7f1b75589155 - std::sys_common::backtrace::print::hfb7ed7d7ba09efbc
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/std/src/sys_common/backtrace.rs:34:9
   8:     0x7f1b7558b19e - std::panicking::default_hook::{{closure}}::h1b562f555ed8e3e1
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/std/src/panicking.rs:271:22
   9:     0x7f1b7558af45 - std::panicking::default_hook::hc7a03bfdd14960de
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/std/src/panicking.rs:290:9
  10:     0x7f1b7558b75e - std::panicking::rust_panic_with_hook::hfd9594253b2174d9
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/std/src/panicking.rs:692:13
  11:     0x7f1b7558b659 - std::panicking::begin_panic_handler::{{closure}}::h0cbfa1758f851517
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/std/src/panicking.rs:583:13
  12:     0x7f1b755897f6 - std::sys_common::backtrace::__rust_end_short_backtrace::ha6a4a6b805e2aab8
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/std/src/sys_common/backtrace.rs:150:18
  13:     0x7f1b7558b3b2 - rust_begin_unwind
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/std/src/panicking.rs:579:5
  14:     0x7f1b755b5da3 - core::panicking::panic_fmt::h46ad3c1c4689a685
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/core/src/panicking.rs:67:14
  15:     0x7f1b755b5b43 - core::panicking::panic_display::h627921064211ab5a
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/core/src/panicking.rs:150:5
  16:     0x7f1b755b5b43 - core::panicking::panic_str::h6d211e3df342c52d
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/core/src/panicking.rs:134:5
  17:     0x7f1b755b5b43 - core::option::expect_failed::hde2939257a72b493
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/core/src/option.rs:2055:5
  18:     0x7f1b74c30638 - cargo[161ae918bb722f79]::core::compiler::unit_dependencies::build_unit_dependencies
  19:     0x7f1b74e7dfa9 - cargo[161ae918bb722f79]::ops::cargo_compile::create_bcx
  20:     0x7f1b74e7bd19 - cargo[161ae918bb722f79]::ops::cargo_compile::compile_ws
  21:     0x7f1b74e7bc08 - cargo[161ae918bb722f79]::ops::cargo_compile::compile
  22:     0x7f1b748d4be5 - cargo[b6922b4ea0bc1e6c]::commands::build::exec
  23:     0x7f1b748ec56a - cargo[b6922b4ea0bc1e6c]::cli::main
  24:     0x7f1b749490cd - cargo[b6922b4ea0bc1e6c]::main
  25:     0x7f1b748e9443 - std[1bf61b140f607083]::sys_common::backtrace::__rust_begin_short_backtrace::<fn(), ()>
  26:     0x7f1b748c71b9 - std[1bf61b140f607083]::rt::lang_start::<()>::{closure#0}
  27:     0x7f1b7557926e - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::hd10c815396418599
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/core/src/ops/function.rs:287:13
  28:     0x7f1b7557926e - std::panicking::try::do_call::ha76430897d9898fb
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/std/src/panicking.rs:487:40
  29:     0x7f1b7557926e - std::panicking::try::h113ffea0d3ff8f7b
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/std/src/panicking.rs:451:19
  30:     0x7f1b7557926e - std::panic::catch_unwind::h776e71b7812e3d7f
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/std/src/panic.rs:140:14
  31:     0x7f1b7557926e - std::rt::lang_start_internal::{{closure}}::h7e12587c0c00e848
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/std/src/rt.rs:148:48
  32:     0x7f1b7557926e - std::panicking::try::do_call::h3159645e6e500b07
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/std/src/panicking.rs:487:40
  33:     0x7f1b7557926e - std::panicking::try::ha595859fd6091cf4
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/std/src/panicking.rs:451:19
  34:     0x7f1b7557926e - std::panic::catch_unwind::h0f722ac499145178
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/std/src/panic.rs:140:14
  35:     0x7f1b7557926e - std::rt::lang_start_internal::hbb924b6e016e922a
                               at /rustc/0c61c7a978fe9f7b77a1d667c77d2202dadd1c10/library/std/src/rt.rs:148:20
  36:     0x7f1b7494bd25 - main
  37:     0x7f1b74009d90 - __libc_start_call_main
                               at ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
  38:     0x7f1b74009e40 - __libc_start_main_impl
                               at ./csu/../csu/libc-start.c:392:3
  39:     0x7f1b748c0029 - <unknown>
  40:                0x0 - <unknown>

Steps

1. Check out the reproduction case at https://github.com/Qix-/cargo-crash-repro-2023-3-28
2. Run the command specified in the README (you might need to change --target to your host's triple, not sure)

Possible Solution(s)

No response

Notes

No response

Version

cargo 1.70.0-nightly (15d090969 2023-03-21)
release: 1.70.0-nightly
commit-hash: 15d090969743630bff549a1b068bcaa8174e5ee3
commit-date: 2023-03-21
host: x86_64-unknown-linux-gnu
libgit2: 1.5.0 (sys:0.16.0 vendored)
libcurl: 8.0.1-DEV (sys:0.4.61+curl-8.0.1 vendored ssl:OpenSSL/1.1.1q)
os: Ubuntu 22.04 (jammy) [64-bit]

[weihanglo]

Might be a bit related I guess: #10444.

[Qix-]

Interestingly I was just experimenting with the bindeps feature but this repro doesn't actually use it in any capacity (unless I missed something). Good catch though, thanks :)

[weihanglo]

Sorry I didn't really look deeply into this issue. Just got the impression that there are some more panics around target platforms interacting with -Zbuild-std, -Zper-package-target, -Zbindeps and their combinations 😆.

[Qix-]

Definitely looks like the same root cause, just manifesting in a different situation in this case. Feel free to close as a dupe if that makes sense :) I appreciate the prompt response!

[Qix-]

Forgot I made this issue; I have some more information here: #9406 (comment)

[Qix-]

Actually, I think I might have found the issue.

If I'm understanding correctly, the std roots that are available for attachment (and are later attached based on the force_target setting) are enumerated beforehand. However, those don't include any additional force_target values that may be present.

Thus, the explicit_host_kinds only includes what's passed as --target here:

cargo/src/cargo/ops/cargo_compile/mod.rs

Lines 345 to 352 in d7d48e4

	let explicit_host_kinds: Vec<_> = build_config
	.requested_kinds
	.iter()
	.map(|kind| match kind {
	CompileKind::Host => explicit_host_kind,
	CompileKind::Target(t) => CompileKind::Target(*t),
	})
	.collect();

If that can somehow be extended to include any additional force_target values, then this should be resolved. I might be out of my depth here, though.

EDIT: Sure enough, it only includes what's on --target.

BUILD CONFIG: BuildConfig {
    requested_kinds: [
        Target(
            CompileTarget {
                name: "x86_64-unknown-linux-gnu",
            },
        ),
    ],
<snip>

[Qix-]

Man, yeah. This feature is really half-baked, it seems :(

I went ahead based on my last findings and did the following:

diff --git a/src/cargo/ops/cargo_compile/mod.rs b/src/cargo/ops/cargo_compile/mod.rs
index 04ebe7d..bfb36f5 100644
--- a/src/cargo/ops/cargo_compile/mod.rs
+++ b/src/cargo/ops/cargo_compile/mod.rs
@@ -342,7 +342,7 @@ pub fn create_bcx<'a, 'cfg>(
     // assuming `--target $HOST` was specified. See
     // `rebuild_unit_graph_shared` for more on why this is done.
     let explicit_host_kind = CompileKind::Target(CompileTarget::new(&target_data.rustc.host)?);
-    let explicit_host_kinds: Vec<_> = build_config
+    let mut explicit_host_kinds: Vec<_> = build_config
         .requested_kinds
         .iter()
         .map(|kind| match kind {
@@ -351,6 +351,9 @@ pub fn create_bcx<'a, 'cfg>(
         })
         .collect();

+    explicit_host_kinds.push(CompileKind::Target(CompileTarget::new("<snip>/triple/x86_64.json").unwrap()));
+    explicit_host_kinds.push(CompileKind::Target(CompileTarget::new("<snip>/triple/x86.json").unwrap()));
+
     // Passing `build_config.requested_kinds` instead of
     // `explicit_host_kinds` here so that `generate_root_units` can do
     // its own special handling of `CompileKind::Host`. It will

It did proceed to compile, but then Cargo detected different target types and blew up:

   Compiling core v0.0.0 (<home>/.rustup/toolchains/nightly-2024-02-06-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core)
   Compiling kernel v0.1.0 (/src/qix-/cargo-crash-repro-2023-3-28/kernel)
error[E0514]: found crate `core` compiled by an incompatible version of rustc
  |
  = note: the following crate versions were found:
          crate `core` compiled by rustc 1.78.0-nightly (256b6fb19 2024-02-06): /src/qix-/cargo-crash-repro-2023-3-28/target/x86_64-unknown-linux-gnu/debug/deps/libcore-254b90bd980a29e6.rmeta
  = help: please recompile that crate using this compiler (rustc 1.78.0-nightly (f067fd608 2024-02-05)) (consider running `cargo clean` first)

error[E0514]: found crate `compiler_builtins` compiled by an incompatible version of rustc
  |
  = note: the following crate versions were found:
          crate `compiler_builtins` compiled by rustc 1.78.0-nightly (256b6fb19 2024-02-06): /src/qix-/cargo-crash-repro-2023-3-28/target/x86_64-unknown-linux-gnu/debug/deps/libcompiler_builtins-71a7bf568f4ad98c.rmeta
  = help: please recompile that crate using this compiler (rustc 1.78.0-nightly (f067fd608 2024-02-05)) (consider running `cargo clean` first)

error: requires `sized` lang_item

For more information about this error, try `rustc --explain E0514`.
error: could not compile `kernel` (lib) due to 3 previous errors
warning: build failed, waiting for other jobs to finish...
error: data-layout for target `x86_64-4136372502990422726`, `e-m:e-i64:64-f80:128-n8:16:32:64-S128`, differs from LLVM target's `x86_64-unknown-myos` default layout, `e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128`

error: could not compile `core` (lib) due to 1 previous error

Seems Cargo just isn't set up for multi-target building like this? I think a proper implementation of this feature is going to require a lot more than a few clever TOML manifest additions :/

This is where I'm going to leave it for now. Hopefully someone much more knowledgeable with the codebase than I am can figure this out. For now, it seems that force_target is unusable for anything other than the most trivial of cases.