Skip to content

CI: add TSAN to the sanitizers pipelines #42444

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Oct 4, 2021
Merged

CI: add TSAN to the sanitizers pipelines #42444

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
9913683
Setup CI for TSAN
tkf Oct 1, 2021
6ed9b7f
Don't build sysimage with TSAN
tkf Oct 1, 2021
a09febb
in the Buildkite cache, don't cache the "registries" directory in the…
tkf Oct 2, 2021
b92db8b
Merge branch 'master' into tkf/tsan-ci
tkf Oct 3, 2021
f22d188
We only run the `tsan` job on Julia 1.8 and later
tkf Oct 3, 2021
ee11eb4
Use build target julia-src-debug
tkf Oct 4, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions .buildkite/pipelines/experimental/misc/sanitizers.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,3 +23,21 @@ steps:
commands: |
echo "--- Build julia-debug with ASAN"
contrib/asan/build.sh ./tmp/test-asan -j$${JULIA_CPU_THREADS:?} debug
- label: "tsan"
key: "tsan"
plugins:
- JuliaCI/julia#v1:
version: 1.6
- staticfloat/sandbox#v1:
rootfs_url: https://github.com/JuliaCI/rootfs-images/releases/download/v3.1/llvm_passes.x86_64.tar.gz
rootfs_treehash: "9dd715500b117a16fcfa419ea0bca0c0ca902cee"
uid: 1000
gid: 1000
workspaces:
- "/cache/repos:/cache/repos"
timeout_in_minutes: 120
commands: |
echo "--- Build julia-debug runtime with TSAN"
contrib/tsan/build.sh ./tmp/test-tsan -j$${JULIA_CPU_THREADS:?} -C src debug
echo "--- BROKEN: Build julia-debug with TSAN"
! make -C ./tmp/test-tsan/tsan -j$${JULIA_CPU_THREADS:?} debug
16 changes: 16 additions & 0 deletions contrib/tsan/Make.user.tsan
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
TOOLCHAIN=$(BUILDROOT)/../toolchain
BINDIR=$(TOOLCHAIN)/usr/bin
TOOLDIR=$(TOOLCHAIN)/usr/tools

# use our new toolchain
USECLANG=1
override CC=$(BINDIR)/clang
override CXX=$(TOOLDIR)/clang++

USE_BINARYBUILDER_LLVM=1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't work. You need to build LLVM with tsan instrumentation since it uses locks.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we still use a normal-build LLVM with deadlock:llvm in the TSAN_OPTIONS=suppressions=... configuration? Or there's no way to work around this fundamentally?

If we need LLVM with TSAN, I guess we'd need to start adding it in BB before trying this in CI. But it'd be a much bigger project than this quick PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I don't want to build LLVM from source in the Base Julia CI. We'd need to have an "LLVM with TSAN" JLL available from Ygg.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we still use a normal-build LLVM with deadlock:llvm in the TSAN_OPTIONS=suppressions=... configuration? Or there's no way to work around this fundamentally?

It might be possible, but I'd probably recommend against it. I don't know if that can introduce false positives.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, so maybe it makes sense to close this PR until a JLL is ready.

Or, maybe we can still just run the build process (-C src), so that some rudimentary bugs can be caught?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a LLVM_tsan_jll makes the most sense. Then it only needs to be built very occasionally in Ygg, instead of building LLVM from source on every PR that is opened to JuliaLang/julia.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really want to support "yet" another version of LLVM in Yggdrasil and slow the update cycle down even more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm. That's a fair point. Let's see if @staticfloat has any ideas.

At least we can keep the first part, which doesn't require building LLVM from source.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the source of slowness in yet another LLVM JLL vs ccache? Aren't they equivalent in terms of machine time? Is it that, with the JLL approach, there need to be humans involved in the process waiting for the JLL and updating the URL/version/checksum? If that's the case, it can also potentially be solved by more automation?

I kinda liked the JLL approach since it'd make it easier for more casual users to try sanitizers. OTOH, I guess the ccache approach would be more hackable for core devs since it'd give us a "healthy" CI'ed build script that is easy to flip some LLVM build options. Maybe the upside of the ccache approach wins, since there's no single set of build options that covers all the needs when you debugging things.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it that, with the JLL approach, there need to be humans involved in the process waiting for the JLL and updating the URL/version/checksum? If that's the case, it can also potentially be solved by more automation?

Yes, the maintenance of the LLVM JLL is a lot of human time, some of it waiting, some of it making sure that all the version number bits line up. Sheparding things through to the registry.

I made attempts at trying to automate it, but currently it is pareto optimal in that I find is really annoying, but not annoying enough to descend to arcane wizardy. c.f JuliaPackaging/Yggdrasil#726 (comment)


override SANITIZE=1
override SANITIZE_THREAD=1

# default to a debug build for better line number reporting
override JULIA_BUILD_MODE=debug
61 changes: 61 additions & 0 deletions contrib/tsan/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
#!/bin/bash
# This file is a part of Julia. License is MIT: https://julialang.org/license

#
# Usage:
# contrib/tsan/build.sh <path> [<make_targets>...]
#
# Build TSAN-enabled julia. Given a workspace directory <path>, build
# TSAN-enabled julia in <path>/tsan. Required toolss are install under
# <path>/toolchain. Note that the same <path> passed to `contrib/asan/build.sh`
# can be used to share the toolchain used for ASAN. This scripts also takes
# optional <make_targets> arguments which are passed to `make`. The default
# make target is `debug`.

set -ue

# `$WORKSPACE` is a directory in which we create `toolchain` and `tsan`
# sub-directories.
WORKSPACE="$1"
shift
if [ "$WORKSPACE" = "" ]; then
echo "Workspace directory must be specified as the first argument" >&2
exit 2
fi

mkdir -pv "$WORKSPACE"
WORKSPACE="$(cd "$WORKSPACE" && pwd)"
if [ "$WORKSPACE" = "" ]; then
echo "Failed to create the workspace directory." >&2
exit 2
fi

HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
JULIA_HOME="$HERE/../../"

echo
echo "Installing toolchain..."

TOOLCHAIN="$WORKSPACE/toolchain"
if [ ! -d "$TOOLCHAIN" ]; then
make -C "$JULIA_HOME" configure O=$TOOLCHAIN
cp "$HERE/../asan/Make.user.tools" "$TOOLCHAIN/Make.user"
fi

make -C "$TOOLCHAIN/deps" install-clang install-llvm-tools

# TODO: https://github.com/JuliaPackaging/Yggdrasil/issues/3359
rm "$TOOLCHAIN/usr/tools/clang++"
ln -s "$TOOLCHAIN/usr/bin/clang" "$TOOLCHAIN/usr/tools/clang++"

echo
echo "Building Julia..."

BUILD="$WORKSPACE/tsan"
if [ ! -d "$BUILD" ]; then
make -C "$JULIA_HOME" configure O="$BUILD"
cp "$HERE/Make.user.tsan" "$BUILD/Make.user"
fi

cd "$BUILD" # so that we can pass `-C src` to `make`
make "$@"