Locally failing test runs

[djc]

When I run cargo test --features test locally on my macOS laptop, I see a lot of failures:

failures:
    suite::cli_self_upd::update_exact
    suite::cli_self_upd::update_no_change
    suite::cli_self_upd::update_precise
    suite::cli_self_upd::updater_leaves_itself_for_later_deletion
    suite::cli_ui::rustup_ui_doc_text_tests
    suite::cli_v1::change_override
    suite::cli_v1::default_existing_toolchain
    suite::cli_v1::expected_bins_exist
    suite::cli_v1::install_override_toolchain_from_archive
    suite::cli_v1::install_override_toolchain_from_channel
    suite::cli_v1::install_override_toolchain_from_version
    suite::cli_v1::install_toolchain_from_archive
    suite::cli_v1::install_toolchain_from_channel
    suite::cli_v1::install_toolchain_from_version
    suite::cli_v1::list_toolchains
    suite::cli_v1::multiple_overrides
    suite::cli_v1::no_update_on_channel_when_date_has_not_changed
    suite::cli_v1::override_overrides_default
    suite::cli_v1::remove_override_no_default
    suite::cli_v1::remove_override_toolchain_err_handling
    suite::cli_v1::remove_override_with_default
    suite::cli_v1::remove_override_with_multiple_overrides
    suite::cli_v1::run_command
    suite::cli_v1::rustc_no_default_toolchain
    suite::cli_v1::update_channel
    suite::cli_v1::update_on_channel_when_date_has_changed
    suite::cli_v2::add_remove_multiple_toolchains
    suite::cli_v2::add_target_v1_toolchain
    suite::cli_v2::bad_manifest
    suite::cli_v2::change_override
    suite::cli_v2::default_existing_toolchain
    suite::cli_v2::expected_bins_exist
    suite::cli_v2::file_override_toolchain_err_handling
    suite::cli_v2::install_allow_downgrade
    suite::cli_v2::install_override_toolchain_from_archive
    suite::cli_v2::install_override_toolchain_from_channel
    suite::cli_v2::install_override_toolchain_from_version
    suite::cli_v2::install_toolchain_from_archive
    suite::cli_v2::install_toolchain_from_channel
    suite::cli_v2::install_toolchain_from_version
    suite::cli_v2::install_with_profile
    suite::cli_v2::list_targets_v1_toolchain
    suite::cli_v2::list_toolchains
    suite::cli_v2::list_toolchains_with_bogus_file
    suite::cli_v2::multiple_overrides
    suite::cli_v2::no_update_on_channel_when_date_has_not_changed
    suite::cli_v2::override_overrides_default
    suite::cli_v2::plus_override_toolchain_err_handling
    suite::cli_v2::remove_override_no_default
    suite::cli_v2::remove_override_toolchain_err_handling
    suite::cli_v2::remove_override_with_default
    suite::cli_v2::remove_override_with_multiple_overrides
    suite::cli_v2::remove_target_missing_update_hash
    suite::cli_v2::remove_toolchain_default
    suite::cli_v2::remove_toolchain_ignore_trailing_slash
    suite::cli_v2::run_command
    suite::cli_v2::run_with_install_flag_against_unavailable_component
    suite::cli_v2::rustc_no_default_toolchain
    suite::cli_v2::target_list_ignores_unavailable_targets
    suite::cli_v2::update_channel
    suite::cli_v2::update_on_channel_when_date_has_changed
    suite::cli_v2::update_removed_component_toolchain
    suite::cli_v2::update_unavailable_force
    suite::cli_v2::update_unavailable_std
    suite::cli_v2::upgrade_v1_to_v2
    suite::cli_v2::upgrade_v2_to_v1

test result: FAILED. 347 passed; 66 failed; 1 ignored; 0 measured; 0 filtered out; finished in 151.65s

However, when I run some of these by themselves, they pass:

djc-2021 master rustup $ cargo test --features test run_command
   Compiling rustup v1.28.2 (/Users/djc/src/rustup)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3.35s
     Running unittests src/lib.rs (target/debug/deps/rustup-f31d68d4705b8022)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 120 filtered out; finished in 0.00s

     Running unittests src/bin/rustup-init.rs (target/debug/deps/rustup_init-5c22cd4f46ee5697)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/test_bonanza.rs (target/debug/deps/test_bonanza-733d924557f73b60)

running 2 tests
test suite::cli_v1::run_command ... ok
test suite::cli_v2::run_command ... ok

test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 412 filtered out; finished in 4.88s

I tried this while checking the 1.27.1 tag, but it worked the same. I'm pretty sure I've had passing local test runs since 1.27.1 was released...

Example failure:

---- suite::cli_v2::run_command stdout ----
ran: rustup ["set", "default-host", "aarch64-apple-darwin"]
inprocess: true
status: Some(0)
duration: 0.014s
stdout:
====

====

stderr:
====

====

ran: rustup ["set", "auto-self-update", "disable"]
inprocess: true
status: Some(0)
duration: 0.006s
stdout:
====

====

stderr:
====
info: auto-self-update mode set to 'disable'

====

ran: rustup ["update", "nightly"]
inprocess: false
status: Some(0)
duration: 0.006s
stdout:
====

====

stderr:
====

====

ran: rustup ["default", "beta"]
inprocess: true
status: Some(0)
duration: 2.014s
stdout:
====

  beta-aarch64-apple-darwin installed - 1.2.0 (hash-beta-1.2.0)


====

stderr:
====
info: syncing channel updates for 'beta-aarch64-apple-darwin'
info: latest update on 2015-01-02, rust version 1.2.0 (hash-beta-1.2.0)
info: downloading component 'cargo'
info: downloading component 'rust-docs'
info: downloading component 'rust-std'
info: downloading component 'rustc'
info: installing component 'cargo'
info: installing component 'rust-docs'
info: installing component 'rust-std'
info: installing component 'rustc'
info: default toolchain set to 'beta-aarch64-apple-darwin'

====

ran: rustup ["run", "nightly", "rustc", "--version"]
inprocess: false
status: Some(0)
duration: 0.008s
stdout:
====

====

stderr:
====

====


thread 'suite::cli_v2::run_command' panicked at /Users/djc/src/rustup/src/test/clitools.rs:180:9:

---- expected: tests/suite/cli_v2.rs:1074:22
++++ actual:   In-memory
   1      - 1.3.0 (hash-nightly-2)

Update with SNAPSHOTS=overwrite

Any suggestions for what might be happening here? This is definitely making it harder to locally debug real test failures.

[ChrisDenton]

I don't know but when I've had trouble with tests before it was because the test suite wasn't resetting the environment as much as it should have been. So maybe try running in as a pristine environment as you can. E.g. with a minimum set of environment variables and a minimal PATH.

[rami3l]

@djc This is very weird because I just refactored all these tests, and of course I ran all of them in batches. Yes, I use an ARM64 Mac as you already know 🤔

I could be helpful if you can figure out what is causing the problem: I'd suspect it was env variables.

[djc]

There doesn't seem to be anything crazy in my env -- nothing RUSTUP_* in any case. I have a fairly extensive PATH though, with /Users/djc/.cargo/bin appearing twice. That seems the only part that's rustup-relevant though...

[rami3l]

@djc I just did another full test suite run on my machine again and 0 failed as usual 😓

Looking at your diff, I thinks the fake rustc somehow got retcode=0 while not outputting anything? Could there be a concurrency bug (🤦‍♀️) in our test suite regarding the creation of fake binaries (in mock_bin.rs)?

[djc]

Could there be a concurrency bug (🤦‍♀️) in our test suite regarding the creation of fake binaries (in mock_bin.rs)?

Yeah, that's what I was thinking as well, but not sure how to further debug that.