0.13.0: cargo-maelstrom `empty_cargo_project`, `few_tests`, `failed_test` fail

[dvzrv]

Specific Maelstrom Program?
cargo-maelstrom

Bug Description
When packaging 0.13.0 for Arch Linux I ran into three test failures for cargo-maelstrom.
empty_cargo_project, few_tests and failed_test seem to fail due to locking issues and /proc mount issues.
(Note, the latter may be due to the restricted build environment we use - a systemd-nspawn container).

If these (integration) tests, should maybe not be run, would it be possible to hide them behind a feature?

How to Reproduce

cargo test --all-features --package cargo-maelstrom

Expected Behavior
All tests pass.

Additional Context

running 3 tests
test empty_cargo_project ... FAILED
test few_tests ... FAILED
test failed_test ... FAILED

failures:

---- empty_cargo_project stdout ----
client process: Mar 11 08:05:34.152 INFO listening on unix-abstract:abd6f
Mar 11 08:05:34.152 DEBG opening log stream
client process: Mar 11 08:05:34.152 INFO got connection, address: (unnamed)
Mar 11 08:05:34.156 DEBG client sending start, request: StartRequest { broker_addr: None, project_dir: "/tmp/.tmpT05oIL/project", state_dir: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/maelstrom/state", container_image_depot_dir: ".cache
/maelstrom/container", cache_dir: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/maelstrom/cache", cache_size: 1000.0 MB, inline_limit: 1000.0 KB, slots: 32, accept_invalid_remote_container_tls_certs: true, artifact_transfer_strategy: TcpUp
load }
Mar 11 08:05:34.156 DEBG [client-process] client starting, slots: 32, inline_limit: 1000.0 KB, cache_size: 1000.0 MB, container_image_depot_cache_dir: ".cache/maelstrom/container", cache_dir: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/m
aelstrom/cache", state_dir: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/maelstrom/state", project_dir: "/tmp/.tmpT05oIL/project", broker_addr: None
Mar 11 08:05:34.156 DEBG [client-process] checking open file limit, estimate: 1597, limit: 4096
Mar 11 08:05:34.165 DEBG dropping Client: waiting for stop request
Mar 11 08:05:34.165 DEBG dropping Client: error waiting for stop: status: Cancelled, message: "operation was canceled", details: [], metadata: MetadataMap { headers: {} }
Mar 11 08:05:34.165 DEBG Client::drop: waiting for dispatcher
client process: Mar 11 08:05:34.166 INFO shutting down, result: Ok(ExitCode(Success))
Mar 11 08:05:34.166 DEBG waiting for child client process, pid: Pid(29816)
Mar 11 08:05:34.168 DEBG UI thread joined

thread 'empty_cargo_project' panicked at crates/cargo-maelstrom/tests/integration_test.rs:93:6:
called `Result::unwrap()` on an `Err` value: Client process error:
creating cache

Caused by:
    lock file "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/maelstrom/cache/local-worker/artifacts/lock" is held by a different process
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- few_tests stdout ----
client process: Mar 11 08:05:34.152 INFO listening on unix-abstract:3e3fe
client process: Mar 11 08:05:34.152 INFO got connection, address: (unnamed)
Mar 11 08:05:34.152 DEBG opening log stream
Mar 11 08:05:34.156 DEBG client sending start, request: StartRequest { broker_addr: None, project_dir: "/tmp/.tmpitFTtp/project", state_dir: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/maelstrom/state", container_image_depot_dir: ".cache
/maelstrom/container", cache_dir: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/maelstrom/cache", cache_size: 1000.0 MB, inline_limit: 1000.0 KB, slots: 32, accept_invalid_remote_container_tls_certs: true, artifact_transfer_strategy: TcpUp
load }
Mar 11 08:05:34.156 DEBG [client-process] client starting, slots: 32, inline_limit: 1000.0 KB, cache_size: 1000.0 MB, container_image_depot_cache_dir: ".cache/maelstrom/container", cache_dir: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/m
aelstrom/cache", state_dir: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/maelstrom/state", project_dir: "/tmp/.tmpitFTtp/project", broker_addr: None
Mar 11 08:05:34.156 DEBG [client-process] checking open file limit, estimate: 1597, limit: 4096
Mar 11 08:05:34.164 DEBG dropping Client: waiting for stop request
Mar 11 08:05:34.165 DEBG dropping Client: error waiting for stop: status: Cancelled, message: "operation was canceled", details: [], metadata: MetadataMap { headers: {} }
Mar 11 08:05:34.165 DEBG Client::drop: waiting for dispatcher
client process: Mar 11 08:05:34.166 INFO shutting down, result: Ok(ExitCode(Success))
Mar 11 08:05:34.166 DEBG waiting for child client process, pid: Pid(29815)
Mar 11 08:05:34.168 DEBG UI thread joined

thread 'few_tests' panicked at crates/cargo-maelstrom/tests/integration_test.rs:93:6:
called `Result::unwrap()` on an `Err` value: Client process error:
creating cache

Caused by:
    lock file "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/maelstrom/cache/local-worker/artifacts/lock" is held by a different process

---- failed_test stdout ----
client process: Mar 11 08:05:34.152 INFO listening on unix-abstract:9c09f
client process: Mar 11 08:05:34.152 INFO got connection, address: (unnamed)
Mar 11 08:05:34.152 DEBG opening log stream
Mar 11 08:05:34.156 DEBG client sending start, request: StartRequest { broker_addr: None, project_dir: "/tmp/.tmpSPwpv6/project", state_dir: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/maelstrom/state", container_image_depot_dir: ".cache
/maelstrom/container", cache_dir: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/maelstrom/cache", cache_size: 1000.0 MB, inline_limit: 1000.0 KB, slots: 32, accept_invalid_remote_container_tls_certs: true, artifact_transfer_strategy: TcpUp
load }
Mar 11 08:05:34.156 DEBG [client-process] client starting, slots: 32, inline_limit: 1000.0 KB, cache_size: 1000.0 MB, container_image_depot_cache_dir: ".cache/maelstrom/container", cache_dir: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/m
aelstrom/cache", state_dir: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/maelstrom/state", project_dir: "/tmp/.tmpSPwpv6/project", broker_addr: None
Mar 11 08:05:34.156 DEBG [client-process] checking open file limit, estimate: 1597, limit: 4096
Mar 11 08:05:34.164 DEBG [client-process] cache initialized, byte_used_target: 1000.0 MB, bytes_used: 0 B, entries: 0, task: dispatcher
Mar 11 08:05:34.164 DEBG [client-process] client started successfully
Mar 11 08:05:34.164 DEBG client completed start
Mar 11 08:05:34.168 DEBG cargo test artifact stream packages, packages: {PackageId { repr: "path+file:///tmp/.tmpSPwpv6/project#0.1.0" }: PackageToBuild { unbuilt_non_test_binaries: {"project"}, built_tests: [] }}
Mar 11 08:05:34.278 DEBG [client-process] run_job, spec: JobSpec { container: ContainerSpec { parent: None, layers: [Stubs(StubsLayerSpec { stubs: ["/{proc,sys,tmp}/", "/dev/{full,null,random,urandom,zero}"] }), Paths(PathsLayerSpec { paths: ["/build/mael
strom/src/maelstrom/crates/cargo-maelstrom/target/debug/deps/project-84bd9c7002fc2c27"], prefix_options: PrefixOptions { strip_prefix: Some("/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/debug/deps"), prepend_prefix: None, canonicalize: fal
se, follow_symlinks: false } }), SharedLibraryDependencies(SharedLibraryDependenciesLayerSpec { binary_paths: ["/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/debug/deps/project-84bd9c7002fc2c27"], prefix_options: PrefixOptions { strip_prefi
x: None, prepend_prefix: None, canonicalize: false, follow_symlinks: true } })], environment: [EnvironmentSpec { vars: {"RUST_BACKTRACE": "$env{RUST_BACKTRACE:-0}", "RUST_LIB_BACKTRACE": "$env{RUST_LIB_BACKTRACE:-0}"}, extend: true }], working_directory:
None, enable_writable_file_system: None, mounts: [Tmp { mount_point: "/tmp" }, Proc { mount_point: "/proc" }, Sys { mount_point: "/sys" }, Devices { devices: EnumSet(Full | Null | Random | Urandom | Zero) }], network: None, user: None, group: None }, prog
ram: "/project-84bd9c7002fc2c27", arguments: ["--exact", "--nocapture", "foo"], timeout: None, estimated_duration: None, allocate_tty: None, priority: 1, capture_file_system_changes: None }
Mar 11 08:05:34.278 DEBG [client-process] add_artifact, path: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/debug/deps/project-84bd9c7002fc2c27"
Mar 11 08:05:34.279 DEBG [client-process] add_artifact, path: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/debug/deps/project-84bd9c7002fc2c27"
Mar 11 08:05:34.279 DEBG [client-process] add_artifact, path: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/maelstrom/cache/manifests/stubs/61cf7d289cb1df8f01604d5a4663e0f133f02748831722ba4aed735357627705.manifest"
Mar 11 08:05:34.309 DEBG [client-process] add_artifact, path: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/maelstrom/cache/manifests/c6a82560bee3943fb7204a80c4107e5734e294e9889f8a765fb11679bffa1317.manifest"
Mar 11 08:05:34.315 DEBG [client-process] add_artifact, path: "/lib/ld-linux-x86-64.so.2"
Mar 11 08:05:34.322 DEBG [client-process] add_artifact, path: "/lib/libc.so.6"
Mar 11 08:05:34.376 DEBG [client-process] add_artifact, path: "/lib/libgcc_s.so.1"
Mar 11 08:05:34.400 DEBG [client-process] add_artifact, path: "/lib/libm.so.6"
Mar 11 08:05:34.427 DEBG [client-process] add_artifact, path: "/lib64/ld-linux-x86-64.so.2"
Mar 11 08:05:34.427 DEBG [client-process] add_artifact, path: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/maelstrom/cache/manifests/06de5b079bb990e85aafede2424c0818891d836f6f3767661db3a05e8035ebf8.manifest"
Mar 11 08:05:34.429 DEBG [client-process] cache added artifact, byte_used_target: 1000.0 MB, bytes_used: 160 B, file_type: Symlink, entries: 1, artifact_bytes_used: 160 B, key: CacheKey { kind: Blob, digest: Sha256Digest(52e7c32a5dc0321ab5b8703083e07dbf31
69e66d1ee364192790a3a0be738213) }, task: dispatcher
Mar 11 08:05:34.429 DEBG [client-process] cache added artifact, byte_used_target: 1000.0 MB, bytes_used: 326 B, file_type: Symlink, entries: 2, artifact_bytes_used: 166 B, key: CacheKey { kind: Blob, digest: Sha256Digest(7a69ecb474fd872b6b30a0444705ad30f5
681fc9f0052e664ea51f7e2ebbd0f5) }, task: dispatcher
Mar 11 08:05:34.429 DEBG [client-process] cache added artifact, byte_used_target: 1000.0 MB, bytes_used: 486 B, file_type: Symlink, entries: 3, artifact_bytes_used: 160 B, key: CacheKey { kind: Blob, digest: Sha256Digest(586993190dbf181464b2517bc2a8517108
de9b51c84ff8209d3228a3dd730ff6) }, task: dispatcher
Mar 11 08:05:34.429 DEBG [client-process] reading digests from manifest, manifest_path: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/maelstrom/cache/local-worker/artifacts/sha256/blob/52e7c32a5dc0321ab5b8703083e07dbf3169e66d1ee364192790a3
a0be738213", task: dispatcher
Mar 11 08:05:34.429 DEBG [client-process] reading digests from manifest, manifest_path: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/maelstrom/cache/local-worker/artifacts/sha256/blob/586993190dbf181464b2517bc2a8517108de9b51c84ff8209d3228
a3dd730ff6", task: dispatcher
Mar 11 08:05:34.429 DEBG [client-process] reading digests from manifest, manifest_path: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/maelstrom/cache/local-worker/artifacts/sha256/blob/7a69ecb474fd872b6b30a0444705ad30f5681fc9f0052e664ea51f
7e2ebbd0f5", task: dispatcher
Mar 11 08:05:34.429 DEBG [client-process] read digests from manifest, result: Ok({Sha256Digest(1335238faa8a0415cca22b70dea24f30407e35f56b28d0afbf1aa5ec12d2c488)}), task: dispatcher
Mar 11 08:05:34.429 DEBG [client-process] cache added artifact, byte_used_target: 1000.0 MB, bytes_used: 582 B, file_type: Symlink, entries: 4, artifact_bytes_used: 96 B, key: CacheKey { kind: Blob, digest: Sha256Digest(1335238faa8a0415cca22b70dea24f30407
e35f56b28d0afbf1aa5ec12d2c488) }, task: dispatcher
Mar 11 08:05:34.429 DEBG [client-process] building bottom FS layer, task: dispatcher
Mar 11 08:05:34.429 DEBG [client-process] read digests from manifest, result: Ok({Sha256Digest(cbe1281c11607ec934798b23370e2dd2eeac7a1b4e53fe162b1745759b05f97d), Sha256Digest(e0dcbd341c883020ae9460de0bef67f487da14a3be2d81bfc4eb171a32fbcf20), Sha256Digest(
781a7e447d8bbab51420190e5920ba27f4c7209553fece464188b09abe87f2f3), Sha256Digest(9b8d6487cdb2e31305c417916b266c5b248d5d571e16fe7ef6329f1dfaf18998)}), task: dispatcher
Mar 11 08:05:34.429 DEBG [client-process] read digests from manifest, result: Ok({}), task: dispatcher
Mar 11 08:05:34.429 DEBG [client-process] building bottom FS layer, task: dispatcher
Mar 11 08:05:34.429 DEBG [client-process] cache added artifact, byte_used_target: 1000.0 MB, bytes_used: 600 B, file_type: Symlink, entries: 5, artifact_bytes_used: 18 B, key: CacheKey { kind: Blob, digest: Sha256Digest(cbe1281c11607ec934798b23370e2dd2eea
c7a1b4e53fe162b1745759b05f97d) }, task: dispatcher
Mar 11 08:05:34.429 DEBG [client-process] cache added artifact, byte_used_target: 1000.0 MB, bytes_used: 618 B, file_type: Symlink, entries: 6, artifact_bytes_used: 18 B, key: CacheKey { kind: Blob, digest: Sha256Digest(e0dcbd341c883020ae9460de0bef67f487d
a14a3be2d81bfc4eb171a32fbcf20) }, task: dispatcher
Mar 11 08:05:34.429 DEBG [client-process] cache added artifact, byte_used_target: 1000.0 MB, bytes_used: 640 B, file_type: Symlink, entries: 7, artifact_bytes_used: 22 B, key: CacheKey { kind: Blob, digest: Sha256Digest(781a7e447d8bbab51420190e5920ba27f4c
7209553fece464188b09abe87f2f3) }, task: dispatcher
Mar 11 08:05:34.429 DEBG [client-process] cache added artifact, byte_used_target: 1000.0 MB, bytes_used: 669 B, file_type: Symlink, entries: 8, artifact_bytes_used: 29 B, key: CacheKey { kind: Blob, digest: Sha256Digest(9b8d6487cdb2e31305c417916b266c5b248
d5d571e16fe7ef6329f1dfaf18998) }, task: dispatcher
Mar 11 08:05:34.429 DEBG [client-process] building bottom FS layer, task: dispatcher
Mar 11 08:05:34.430 DEBG [client-process] built bottom FS layer, result: Ok((TempDir(TempDir { path: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/maelstrom/cache/local-worker/artifacts/tmp/.tmp8DASXN" }), 293)), task: dispatcher
Mar 11 08:05:34.430 DEBG [client-process] cache added artifact, byte_used_target: 1000.0 MB, bytes_used: 962 B, file_type: Directory, entries: 9, artifact_bytes_used: 293 B, key: CacheKey { kind: BottomFsLayer, digest: Sha256Digest(586993190dbf181464b2517
bc2a8517108de9b51c84ff8209d3228a3dd730ff6) }, task: dispatcher
Mar 11 08:05:34.430 DEBG [client-process] built bottom FS layer, result: Ok((TempDir(TempDir { path: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/maelstrom/cache/local-worker/artifacts/tmp/.tmphxtssR" }), 1210)), task: dispatcher
Mar 11 08:05:34.430 DEBG [client-process] cache added artifact, byte_used_target: 1000.0 MB, bytes_used: 2.2 KB, file_type: Directory, entries: 10, artifact_bytes_used: 1.2 KB, key: CacheKey { kind: BottomFsLayer, digest: Sha256Digest(52e7c32a5dc0321ab5b8
703083e07dbf3169e66d1ee364192790a3a0be738213) }, task: dispatcher
Mar 11 08:05:34.431 DEBG [client-process] built bottom FS layer, result: Ok((TempDir(TempDir { path: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/maelstrom/cache/local-worker/artifacts/tmp/.tmpM13TgU" }), 1179)), task: dispatcher
Mar 11 08:05:34.431 DEBG [client-process] cache added artifact, byte_used_target: 1000.0 MB, bytes_used: 3.4 KB, file_type: Directory, entries: 11, artifact_bytes_used: 1.2 KB, key: CacheKey { kind: BottomFsLayer, digest: Sha256Digest(7a69ecb474fd872b6b30
a0444705ad30f5681fc9f0052e664ea51f7e2ebbd0f5) }, task: dispatcher
Mar 11 08:05:34.431 DEBG [client-process] building upper FS layer, task: dispatcher
Mar 11 08:05:34.432 DEBG [client-process] built upper FS layer, result: Ok((TempDir(TempDir { path: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/maelstrom/cache/local-worker/artifacts/tmp/.tmpZQ051l" }), 717)), task: dispatcher
Mar 11 08:05:34.432 DEBG [client-process] cache added artifact, byte_used_target: 1000.0 MB, bytes_used: 4.1 KB, file_type: Directory, entries: 12, artifact_bytes_used: 717 B, key: CacheKey { kind: UpperFsLayer, digest: Sha256Digest(e96f3131c5c617e422659b
440bd2ef4b5bfb99cfe766412e53171b874e409167) }, task: dispatcher
Mar 11 08:05:34.432 DEBG [client-process] building upper FS layer, task: dispatcher
Mar 11 08:05:34.434 DEBG [client-process] built upper FS layer, result: Ok((TempDir(TempDir { path: "/build/maelstrom/src/maelstrom/crates/cargo-maelstrom/target/maelstrom/cache/local-worker/artifacts/tmp/.tmpBNMXKE" }), 1909)), task: dispatcher
Mar 11 08:05:34.434 DEBG [client-process] cache added artifact, byte_used_target: 1000.0 MB, bytes_used: 6.0 KB, file_type: Directory, entries: 13, artifact_bytes_used: 1.9 KB, key: CacheKey { kind: UpperFsLayer, digest: Sha256Digest(3ea480da3ef0e060ac417
fae0176f0eb300c67fa0a2e557564ef2ebebe795f03) }, task: dispatcher
Mar 11 08:05:34.434 DEBG [client-process] starting job, spec: JobSpec { program: "/project-84bd9c7002fc2c27", arguments: ["--exact", "--nocapture", "foo"], environment: ["RUST_BACKTRACE=0", "RUST_LIB_BACKTRACE=0"], layers: NonEmpty { head: (Sha256Digest(7
a69ecb474fd872b6b30a0444705ad30f5681fc9f0052e664ea51f7e2ebbd0f5), Manifest), tail: [(Sha256Digest(586993190dbf181464b2517bc2a8517108de9b51c84ff8209d3228a3dd730ff6), Manifest), (Sha256Digest(52e7c32a5dc0321ab5b8703083e07dbf3169e66d1ee364192790a3a0be738213)
, Manifest)] }, mounts: [Tmp { mount_point: "/tmp" }, Proc { mount_point: "/proc" }, Sys { mount_point: "/sys" }, Devices { devices: EnumSet(Full | Null | Random | Urandom | Zero) }], network: Disabled, root_overlay: None, working_directory: "/", user: Us
erId(0), group: GroupId(0), timeout: None, estimated_duration: None, allocate_tty: None, priority: 1 }, jid: JobId { cid: ClientId(0), cjid: ClientJobId(0) }, task: dispatcher
Mar 11 08:05:34.435 DEBG [client-process] running FUSE file-system, jid: JobId { cid: ClientId(0), cjid: ClientJobId(0) }, task: dispatcher
Mar 11 08:05:34.435 DEBG dropping Client: waiting for stop request
Mar 11 08:05:34.436 DEBG Client::drop: waiting for dispatcher
Mar 11 08:05:34.437 DEBG waiting for child client process, pid: Pid(29814)
client process: Mar 11 08:05:34.437 INFO shutting down, result: Ok(ExitCode(Success))
Mar 11 08:05:34.439 DEBG UI thread joined

thread 'failed_test' panicked at crates/cargo-maelstrom/tests/integration_test.rs:151:5:
project foo...........................ERR
system error: fsmount for mount of proc: Operation
 not permitted

================== Test Summary ==================
Successful Tests:         0
Failed Tests    :         1
    project foo: failure


failures:
    empty_cargo_project
    failed_test
    few_tests

test result: FAILED. 0 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.32s

error: test failed, to rerun pass `-p cargo-maelstrom --test integration_test`

maelstrom-0.13.0-1-x86_64-build.log
maelstrom-0.13.0-1-x86_64-check.log

[nfachan]

These tests are integration tests.

I can't reproduce locally with my normal setup, but I'll try in a few other environments to see.

This illustrates an advantage of Maelstrom: if these tests were run in Maelstrom, they'd have a well-defined environment. (In our own tests, we always run them in Maelstrom).