How to add executable tool and its environment path to all tests? (sanitizers, symbolizer)

[mikael-s-persson]

Hi!
I'm wondering what the best way is to add an executable tool dependency to all tests (under some build configuration) as well as the corresponding path in an environment variable (that's the easy part).

Context. I'm using toolchains_llvm to get an LLVM toolchain to built my code with, which has been working great for both local fixed-toolchain builds and cross-compilation. Now, I want to be able to build and run my test suite (locally, not cross-compiled, at least for now) under the different sanitizers (AddressSanitizer, MemorySanitizer, ThreadSanitizer, UndefinedSanitizer). So far, this pretty much worked out-of-the-box (with a bit of fiddling with flags). But there are some false-positives that I have to exclude through ignorelists, and the ignorelists require that the relevant sanitized binaries have access to a symbolizer (i.e., "llvm-symbolizer"). The toolchains_llvm provides the symbolizer binary and it is available as a Label in Bazel.

Question. For example, for AddressSanitizer, I need every test (cc_test) to run with the environment variable ASAN_SYMBOLIZER_PATH set to a path to llvm-symbolizer, and of course, have that binary available during execution. I know that I can do this by changing the cc_test rules (or making a macro that wraps it) to add the "data" dependency and set the env-var. But that is very undesirable for many obvious reasons. Is there a way to do it without that? (and without jail-breaking out of the sandbox, obviously ;) )

Some options that I believe might offer solutions (but I seem to be missing pieces):

1. It seems that platform rules are meant to allow this somehow, via things like exec_properties? I couldn't really follow the breadcrumbs through the docs to figure out how this would actually work.
2. It seems that aspects is another potential way to do this. I understand that I could create an aspect that attaches to *_test targets (via ExecutionInfo or RunEnvironmentInfo?), but I'm a bit fuzzy about how to have an aspect that creates an entirely new test suite (with the tool attached) and runs that instead of the normal test suite. And since aspects are pretty laborious to write, I'd rather know before spending too much time on that.

I'm hoping there is a much easier way, and that I've missed something obvious.
Thanks!

[fmeum]

--run_under sounds like a great way to inject a wrapper that 1) sets the environment variables and 2) carries the ignorelists along in its runfiles.

[mikael-s-persson]

The ignorelists can be passed in through the toolchain's extra compiler inputs and corresponding flags, so that's no issue. The only issue with the ignorelists is that they're useless without the symbolizer tool being available.

The --run_under flag at the command-line cannot be used to add a tool dependency to the execution platform. The tool dependency (symbolizer) is a Label formed during the analysis phase, it or its path is not available during the command-line invocation. The --action_env parameter can be used to set an environment variable in general, but the same issue applies, there is no path to the analysis-phase labeled tool that can be used at that point. In other things I have experience with, such tool dependencies can be passed into rules, aspects, platforms or toolchains via hidden attributes to be used in their implementations. So, my question is basically about where and how to route that hidden attribute such that I can inject it into the test execution environment.

I guess a passable solution might be "build" the symbolizer tool (which really only makes a symlink to it in bazel-bin) and then have a wrapper script that use to that path inside a run_under wrapper. That's more or less what I was referring to as "jail-breaking out of the sandbox" in my original post. I would prefer a solution that keeps the build graph hermetic.

[fmeum]

--run_under accepts a label (as well as a shell command) and, at least with a recent rolling release, configures it for the test exec platform (e262d02). Does that work for you?

[mikael-s-persson]

Oh.. Interesting! That works! I did not expect run_under would accept a label.

For the record, this is the basic mechanics of making this work (e.g., for ASan):

In a BUILD file somewhere:

genrule(
    name = "sanitizer_runner_gen",
    srcs = ["@llvm_toolchain//:llvm-symbolizer",],
    outs = ["sanitizer_runner.sh"],
    cmd = "echo -e \"export ASAN_SYMBOLIZER_PATH=$(location @llvm_toolchain//:llvm-symbolizer)\nsetarch x86_64 -R \\\"\\$$@\\\"\" > \"$@\"",
    executable = True,
)

sh_binary(
    name = "sanitizer_runner",
    srcs = [":sanitizer_runner_gen"],
    data = [
        "@llvm_toolchain//:llvm-symbolizer",
    ],
    visibility = ["//visibility:public"],
)

The above just generates a "sanitizer_runner" script that depends on the symbolizer and sets its path in the environment variable.
Note 1: The setarch x86_64 -R is just a required preamble for sanitizer runs that disables ASLR.
Note 2: The symbolizer label must appear in the srcs field of the genrule, not in the tools field so that its path is resolved to the target environment build cache, and not the host environment.

Then, in the command-line or bazelrc file, you pass in --run_under=//some/path:sanitizer_runner.