Spurious rebuilds due to filesystem rounding file modification timestamps

[kornelski]

Problem

The issue #2874 was not completely fixed by #2880. I'm still seeing the exact same problem in latest Cargo:

Dirty libc v0.2.142: the dependency build_script_build was rebuilt (1682215868.647155753s, 647155753ns after last build at 1682215868.000000000s)
Dirty openssl-sys v0.9.86: the dependency build_script_main was rebuilt (1682214777.891768399s, 891768399ns after last build at 1682214777.000000000s)

Steps

I'm using Docker and mounting target/ dir as a Docker volume (in order to cache it). I think the docker volume rounds the timestamps to 1-second precision. However, this doesn't always cause spurious rebuilds, only maybe 25% of the time, and it's always build_script_build or build_script_main, not other files.

Possible Solution(s)

I assume Cargo wouldn't want to introduce imprecision in comparison, such as always comparing the integer number of seconds or allowing 1-second difference between timestamps. Therefore, I propose detecting this problem explicitly: there's only one in a billion chance that the nanosecond part will be exactly 0. So if either of the timestamps being compared has exactly 0 nanosecond part, compare only the integer number of seconds. Otherwise compare in full precision. I can make a PR with that.

Notes

No response

Version

cargo 1.71.0-nightly (d0a4cbcee 2023-04-16)
release: 1.71.0-nightly
commit-hash: d0a4cbcee614fdb7ba66e860e603a00a644d71f8
commit-date: 2023-04-16
host: aarch64-apple-darwin
libgit2: 1.6.3 (sys:0.17.0 vendored)
libcurl: 7.88.1 (sys:0.4.61+curl-8.0.1 system ssl:(SecureTransport) LibreSSL/3.3.6)
ssl: OpenSSL 1.1.1t  7 Feb 2023
os: Mac OS 13.4.0 [64-bit]

[weihanglo]

The proposed solution sounds hacky initially but actually makes some sense.

The precision of mtime timestamp varies in filesystem implementations I guess? I think we need research about what the precision is of each major file system handling timestamp. I know macbook once use 1-second accuracy HFS+ but not APFS is good at nanosecond level. I also wonder if there is any other factor affecting the round-up, perhaps different libc implementations? What if people in a less-accurate filesystem with powerful computing resources, leading to a lot of rebuild skipped.

[Emilgardis]

I'm also seeing this in docker for windows using wsl2

This has actually been something users using cross-rs (which uses a container engine) have dealt with and haven't located the exact reason until now

    Dirty serde_json v1.0.108: the dependency build_script_build was rebuilt (1701461274.529880500s, 529880500ns after last build at 1701461274.000000000s)
Compiling serde_json v1.0.108
    Dirty anyhow v1.0.75: the dependency build_script_build was rebuilt (1701461276.248774900s, 248774900ns after last build at 1701461276.000000000s)

I'm curious if the project would be interested in making the suggested workaround, or if maybe we could enable a way to tell cargo to be less exact in mtime checks (like a cargo flag, available through config)

[kornelski]

@weihanglo I'm unable to perform the research of filesystems and libc implementations that you suggested.

I've been using a fork of Cargo with this change, and it worked fine for me.

If someone else wants to develop a less hacky fix or investigate all platforms, please do.

[weihanglo]

@kornelski, could you share the patch so we can weigh on how we can integrate that, or provide a config for low precision filesystem as a escape hatch. Using fingerprint instead of mtime is also under the consideration #6529. Since we have experimental SQLite integration, #6529 might be more possible to move forward.

[yuyang-ok]

I think I have this issue too.
Sometimes cargo tries to rebuild from libc .

[weihanglo]

Sometimes cargo tries to rebuild from libc .

Hi, @yuyang-ok. could you share what filesystem/OS/environment you are on? That would help evaluate possible solutions.