Regression on x86_64-apple-darwin between 1.0.93 and 1.0.100

[ijackson]

Hi.

Since some time a little over a month ago, we have not been able to build with --target x86_64-apple-darwin. We see this:

The following warnings were emitted during compilation:
warning: zstd-sys@2.0.12+zstd.1.5.6: osxcross: error: cannot find libc++ headers
warning: zstd-sys@2.0.12+zstd.1.5.6: osxcross: error: while detecting target
warning: zstd-sys@2.0.12+zstd.1.5.6: ToolExecError: Command "o64-clang" "-Os" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "--target=x86_64-apple-darwin" "-mmacosx-version-min=10.12" "-I" "zstd/lib/" "-I" "zstd/lib/common" "-march=westmere" "-fvisibility=hidden" "-DZSTD_LIB_DEPRECATED=0" "-DXXH_PRIVATE_API=" "-DZSTDLIB_VISIBILITY=" "-DZSTDERRORLIB_VISIBILITY=" "-o" "/arti/target/x86_64-apple-darwin/release/build/zstd-sys-9cc4c92c9feeab86/out/8957a4c8b028b07b-debug.o" "-c" "zstd/lib/common/debug.c" with args o64-clang did not execute successfully (status code exit status: 1).cargo:warning=osxcross: error: cannot find libc++ headers

The breakage occurs when we try to cargo update. We know that 1.0.93 works, and that 1.0.100 doesn't.

I'm afraid I haven't been able to narrow this down or bisect it properly, because it's in an environment I don't have local access to. The failure occurs for us in CI, with the image amd64/rust:1.76.0-alpine3.18.

If it would be helpful, I could bisect it to a particular cc version.

We do have some more information: logs from a failed build, and an MR in our tree proposing to pin cc, with some discussion.

[ijackson]

Our ticket https://gitlab.torproject.org/tpo/core/arti/-/issues/1507

[NobodyXu]

The only change we introduce in 1.0.94 is for xrun:

1.0.93...1.0.94

We changed --show-sdk-platform-version to --show-sdk-version.

It might be that the xrun you use can't handle that flag?

cc @BlackHoleFox

[BlackHoleFox]

Not sure to start here, apologies. I'm not familiar with how osxcross works, its been years since I've used it myself :(

@ijackson Does the problem go away on cc 1.0.100 if you explicitly set like MACOSX_DEPLOYMENT_TARGET=10.12 in your CI env config? That would assist in narrowing down to xcrun being the problem case since it should outright skip calling it when someone explicitly configures the target version instead.

[ijackson]

Hi. Thanks for looking. Maybe we can blunder through the forest together ? :-)

I ran such a test in our CI (using the gitlab feature to set an env var). It doesn't seem to have made any significant difference, although the logs are gigantic and I'm not sure how I should interpret them.

Test with cc 1.0.11 but not the extra env var

https://gitlab.torproject.org/Diziet/arti/-/jobs/636719

create_symlink x86_64-apple-darwin16-wrapper x86_64h-apple-darwin16-xcodebuild


Given SDK does not contain libc++ headers (-stdlib=libc++ test may fail)
You may want to re-package your SDK using 'tools/gen_sdk_package.sh' on macOS

testing i386-apple-darwin16-clang++ -stdlib=libc++ -std=c++11 ... failed (ignored)
testing x86_64-apple-darwin16-clang++ -stdlib=libc++ -std=c++11 ... failed (ignored)

testing i386-apple-darwin16-clang ... works
testing i386-apple-darwin16-clang++ ... works

testing x86_64h-apple-darwin16-clang ... works
testing x86_64h-apple-darwin16-clang++ ... works

testing x86_64-apple-darwin16-clang ... works
testing x86_64-apple-darwin16-clang++ ... works

Do not forget to add

�[32m/arti/osxcross/target/bin�[0m

to your PATH variable.

All done! Now you can use o32-clang(++) and o64-clang(++) like a normal compiler.

Example usage:

Example 1: CC=o32-clang ./configure --host=i386-apple-darwin16
Example 2: CC=i386-apple-darwin16-clang ./configure --host=i386-apple-darwin16
Example 3: o64-clang -Wall test.c -o test
Example 4: x86_64-apple-darwin16-strip -x test

[... much later ...]
   Compiling libsqlite3-sys v0.28.0
   Compiling matchers v0.1.0
The following warnings were emitted during compilation:

warning: libsqlite3-sys@0.28.0: osxcross: error: cannot find libc++ headers
warning: libsqlite3-sys@0.28.0: osxcross: error: while detecting target

error: failed to run custom build command for `libsqlite3-sys v0.28.0`

Caused by:
  process didn't exit successfully: `/arti/target/release/build/libsqlite3-sys-39dc4def12e47957/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-env-changed=LIBSQLITE3_SYS_USE_PKG_CONFIG
  cargo:include=/usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libsqlite3-sys-0.28.0/sqlite3
  cargo:rerun-if-changed=sqlite3/sqlite3.c
  cargo:rerun-if-changed=sqlite3/wasm32-wasi-vfs.c
  cargo:rerun-if-env-changed=SQLITE_MAX_VARIABLE_NUMBER
  cargo:rerun-if-env-changed=SQLITE_MAX_EXPR_DEPTH
  cargo:rerun-if-env-changed=SQLITE_MAX_COLUMN
  cargo:rerun-if-env-changed=LIBSQLITE3_FLAGS
  TARGET = Some(x86_64-apple-darwin)
  OPT_LEVEL = Some(s)
  HOST = Some(x86_64-unknown-linux-musl)
  cargo:rerun-if-env-changed=CC_x86_64-apple-darwin
  CC_x86_64-apple-darwin = None
  cargo:rerun-if-env-changed=CC_x86_64_apple_darwin
  CC_x86_64_apple_darwin = None
  cargo:rerun-if-env-changed=TARGET_CC
  TARGET_CC = None
  cargo:rerun-if-env-changed=CC
  CC = Some(o64-clang)
  cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some(false)
  cargo:rerun-if-env-changed=CFLAGS_x86_64-apple-darwin
  CFLAGS_x86_64-apple-darwin = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64_apple_darwin
  CFLAGS_x86_64_apple_darwin = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = Some(-march=westmere)
  cargo:warning=osxcross: error: cannot find libc++ headers
  cargo:warning=osxcross: error: while detecting target

  --- stderr


  error occurred: Command "o64-clang" "-Os" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "--target=x86_64-apple-darwin" "-mmacosx-version-min=10.12" "-march=westmere" "-DSQLITE_CORE" "-DSQLITE_DEFAULT_FOREIGN_KEYS=1" "-DSQLITE_ENABLE_API_ARMOR" "-DSQLITE_ENABLE_COLUMN_METADATA" "-DSQLITE_ENABLE_DBSTAT_VTAB" "-DSQLITE_ENABLE_FTS3" "-DSQLITE_ENABLE_FTS3_PARENTHESIS" "-DSQLITE_ENABLE_FTS5" "-DSQLITE_ENABLE_JSON1" "-DSQLITE_ENABLE_LOAD_EXTENSION=1" "-DSQLITE_ENABLE_MEMORY_MANAGEMENT" "-DSQLITE_ENABLE_RTREE" "-DSQLITE_ENABLE_STAT2" "-DSQLITE_ENABLE_STAT4" "-DSQLITE_SOUNDEX" "-DSQLITE_THREADSAFE=1" "-DSQLITE_USE_URI" "-DHAVE_USLEEP=1" "-D_POSIX_THREAD_SAFE_FUNCTIONS" "-DHAVE_ISNAN" "-DHAVE_LOCALTIME_R" "-o" "/arti/target/x86_64-apple-darwin/release/build/libsqlite3-sys-21b89a88b6873b0e/out/0343851c9008f270-sqlite3.o" "-c" "sqlite3/sqlite3.c" with args o64-clang did not execute successfully (status code exit status: 1).


warning: build failed, waiting for other jobs to finish...
The following warnings were emitted during compilation:

warning: lzma-sys@0.1.20: osxcross: error: cannot find libc++ headers
warning: lzma-sys@0.1.20: osxcross: error: while detecting target
warning: lzma-sys@0.1.20: ToolExecError: Command "o64-clang" "-Os" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "--target=x86_64-apple-darwin" "-mmacosx-version-min=10.12" "-I" "xz-5.2/src/liblzma/api" "-I" "xz-5.2/src/liblzma/lzma" "-I" "xz-5.2/src/liblzma/lz" "-I" "xz-5.2/src/liblzma/check" "-I" "xz-5.2/src/liblzma/simple" "-I" "xz-5.2/src/liblzma/delta" "-I" "xz-5.2/src/liblzma/common" "-I" "xz-5.2/src/liblzma/rangecoder" "-I" "xz-5.2/src/common" "-I" "/usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/lzma-sys-0.1.20" "-march=westmere" "-std=c99" "-pthread" "-DHAVE_CONFIG_H=1" "-o" "/arti/target/x86_64-apple-darwin/release/build/lzma-sys-dfbbc672800f8248/out/ebd7929902bc829f-tuklib_cpucores.o" "-c" "xz-5.2/src/common/tuklib_cpucores.c" with args o64-clang did not execute successfully (status code exit status: 1).cargo:warning=osxcross: error: cannot find libc++ headers

Test with cc 1.0.11 and additional MACOSX_DEPLOYMENT_TARGET setting

https://gitlab.torproject.org/Diziet/arti/-/jobs/637107

create_symlink x86_64-apple-darwin16-wrapper x86_64h-apple-darwin16-xcodebuild


Given SDK does not contain libc++ headers (-stdlib=libc++ test may fail)
You may want to re-package your SDK using 'tools/gen_sdk_package.sh' on macOS

testing i386-apple-darwin16-clang++ -stdlib=libc++ -std=c++11 ... failed (ignored)
testing x86_64-apple-darwin16-clang++ -stdlib=libc++ -std=c++11 ... failed (ignored)

testing i386-apple-darwin16-clang ... works
testing i386-apple-darwin16-clang++ ... works

testing x86_64h-apple-darwin16-clang ... works
testing x86_64h-apple-darwin16-clang++ ... works

testing x86_64-apple-darwin16-clang ... works
testing x86_64-apple-darwin16-clang++ ... works

Do not forget to add

�[32m/arti/osxcross/target/bin�[0m

to your PATH variable.

All done! Now you can use o32-clang(++) and o64-clang(++) like a normal compiler.

Example usage:

Example 1: CC=o32-clang ./configure --host=i386-apple-darwin16
Example 2: CC=i386-apple-darwin16-clang ./configure --host=i386-apple-darwin16
Example 3: o64-clang -Wall test.c -o test
Example 4: x86_64-apple-darwin16-strip -x test

[...much later...]
   Compiling libsqlite3-sys v0.28.0
   Compiling matchers v0.1.0
   Compiling proc-macro-crate v1.3.1
   Compiling proc-macro-crate v3.1.0
The following warnings were emitted during compilation:

warning: libsqlite3-sys@0.28.0: Compiler family detection failed due to error: ToolExecError: Command "o64-clang" "-E" "/arti/target/x86_64-apple-darwin/release/build/libsqlite3-sys-21b89a88b6873b0e/out/9613648878214335138detect_compiler_family.c" with args o64-clang did not execute successfully (status code exit status: 1).
warning: libsqlite3-sys@0.28.0: osxcross: error: cannot find libc++ headers
warning: libsqlite3-sys@0.28.0: osxcross: error: while detecting target
warning: libsqlite3-sys@0.28.0: osxcross: error: cannot find libc++ headers
warning: libsqlite3-sys@0.28.0: osxcross: error: while detecting target

error: failed to run custom build command for `libsqlite3-sys v0.28.0`

Caused by:
  process didn't exit successfully: `/arti/target/release/build/libsqlite3-sys-39dc4def12e47957/build-script-build` (exit status: 1)
  --- stdout
  cargo:rerun-if-env-changed=LIBSQLITE3_SYS_USE_PKG_CONFIG
  cargo:include=/usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/libsqlite3-sys-0.28.0/sqlite3
  cargo:rerun-if-changed=sqlite3/sqlite3.c
  cargo:rerun-if-changed=sqlite3/wasm32-wasi-vfs.c
  cargo:rerun-if-env-changed=SQLITE_MAX_VARIABLE_NUMBER
  cargo:rerun-if-env-changed=SQLITE_MAX_EXPR_DEPTH
  cargo:rerun-if-env-changed=SQLITE_MAX_COLUMN
  cargo:rerun-if-env-changed=LIBSQLITE3_FLAGS
  TARGET = Some(x86_64-apple-darwin)
  OPT_LEVEL = Some(s)
  HOST = Some(x86_64-unknown-linux-musl)
  cargo:rerun-if-env-changed=CC_x86_64-apple-darwin
  CC_x86_64-apple-darwin = None
  cargo:rerun-if-env-changed=CC_x86_64_apple_darwin
  CC_x86_64_apple_darwin = None
  cargo:rerun-if-env-changed=TARGET_CC
  TARGET_CC = None
  cargo:rerun-if-env-changed=CC
  CC = Some(o64-clang)
  cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
  cargo:warning=Compiler family detection failed due to error: ToolExecError: Command "o64-clang" "-E" "/arti/target/x86_64-apple-darwin/release/build/libsqlite3-sys-21b89a88b6873b0e/out/9613648878214335138detect_compiler_family.c" with args o64-clang did not execute successfully (status code exit status: 1).
  cargo:warning=osxcross: error: cannot find libc++ headers
  cargo:warning=osxcross: error: while detecting target
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some(false)
  cargo:rerun-if-env-changed=CFLAGS_x86_64-apple-darwin
  CFLAGS_x86_64-apple-darwin = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64_apple_darwin
  CFLAGS_x86_64_apple_darwin = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = Some(-march=westmere)
  cargo:warning=osxcross: error: cannot find libc++ headers
  cargo:warning=osxcross: error: while detecting target

  --- stderr


  error occurred: Command "o64-clang" "-Os" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "--target=x86_64-apple-darwin" "-mmacosx-version-min=10.12" "-march=westmere" "-DSQLITE_CORE" "-DSQLITE_DEFAULT_FOREIGN_KEYS=1" "-DSQLITE_ENABLE_API_ARMOR" "-DSQLITE_ENABLE_COLUMN_METADATA" "-DSQLITE_ENABLE_DBSTAT_VTAB" "-DSQLITE_ENABLE_FTS3" "-DSQLITE_ENABLE_FTS3_PARENTHESIS" "-DSQLITE_ENABLE_FTS5" "-DSQLITE_ENABLE_JSON1" "-DSQLITE_ENABLE_LOAD_EXTENSION=1" "-DSQLITE_ENABLE_MEMORY_MANAGEMENT" "-DSQLITE_ENABLE_RTREE" "-DSQLITE_ENABLE_STAT2" "-DSQLITE_ENABLE_STAT4" "-DSQLITE_SOUNDEX" "-DSQLITE_THREADSAFE=1" "-DSQLITE_USE_URI" "-DHAVE_USLEEP=1" "-D_POSIX_THREAD_SAFE_FUNCTIONS" "-DHAVE_ISNAN" "-DHAVE_LOCALTIME_R" "-o" "/arti/target/x86_64-apple-darwin/release/build/libsqlite3-sys-21b89a88b6873b0e/out/0343851c9008f270-sqlite3.o" "-c" "sqlite3/sqlite3.c" with args o64-clang did not execute successfully (status code exit status: 1).


warning: build failed, waiting for other jobs to finish...
The following warnings were emitted during compilation:

warning: zstd-sys@2.0.12+zstd.1.5.6: Compiler family detection failed due to error: ToolExecError: Command "o64-clang" "-E" "/arti/target/x86_64-apple-darwin/release/build/zstd-sys-c90573e828bfe880/out/4392794065767800547detect_compiler_family.c" with args o64-clang did not execute successfully (status code exit status: 1).
warning: zstd-sys@2.0.12+zstd.1.5.6: osxcross: error: cannot find libc++ headers
warning: zstd-sys@2.0.12+zstd.1.5.6: osxcross: error: while detecting target

[ijackson]

We have discovered that cached data (target/ directories) in our CI would sometimes make this job succeed, when it would otherwise fail. Apparently not everything about the inputs is captured by cargo's cache invalidation algorithm. I think it's likely that the version range I've quoted is wrong.

[NobodyXu]

It might be because we are missing some re-runs for files?