[bazel] add more documentation around our Bazel build

.bazelrc

@@ -65,6 +65,9 @@ build --nobuild_runfile_links
 # common --verbose_explanations
 
 # Compress any artifacts larger than 2MiB with zstd.
+#
+# Note(parkmycar): These thresholds were chosen arbitrarily. You should feel
+# free to change them if you encounter issues.
 common --remote_cache_compression
 common --experimental_remote_cache_compression_threshold=2097152
 # Memoizes merkle tree calculations to improve the remote cache hit checking speed.
@@ -105,7 +108,9 @@ build:linux --linkopt="-Wl,-O2"
 build:linux --@rules_rust//:extra_rustc_flag="-Clink-arg=-Wl,-O2"
 build:linux --copt="-gz=zlib"
 
-# Match the DWARF version used by Rust
+# Match the DWARF version used by Rust.
+#
+# Note(parkmycar): This might not be necessary but seemed nice to do.
 #
 # See: <https://doc.rust-lang.org/stable/unstable-book/compiler-flags/dwarf-version.html>
 build:linux --copt="-gdwarf-4"
@@ -141,7 +146,8 @@ build --@rules_rust//:extra_rustc_flag="-Csymbol-mangling-version=v0"
 # Cargo when incremental compilation is enabled.
 build --@rules_rust//:extra_rustc_flag="-Ccodegen-units=64"
 # Enabling pipelined builds allows dependent libraries to begin compiling with
-# just `.rmeta` instead of the full `.rlib`.
+# just `.rmeta` instead of the full `.rlib`. This is what Cargo does and offers
+# a significant speedup in end-to-end build times!
 build --@rules_rust//rust/settings:pipelined_compilation=True
 
 # `cargo check` like config, still experimental!
@@ -248,19 +254,3 @@ build:hwasan --action_env=ASAN_OPTIONS=verbosity=1
 # HACK(parkmycar): We want to tell Rust to use the Bazel provided `clang++` but there isn't
 # a great way to do that. We know this is where `clang++` lives though, so it works.
 build:hwasan --@rules_rust//:extra_rustc_flag=-Clinker=external/llvm_toolchain_llvm/bin/clang++
-
-# Cross Language LTO
-#
-# <https://blog.llvm.org/2019/09/closing-gap-cross-language-lto-between.html>
-#
-# TODO(parkmycar): Re-enable these. They eiter cause a compilation failure because of
-# missing object files, or (seemingly) makes the number of codegen units 1 which compiles
-# way too slow.
-#build:release --@rules_rust//:extra_rustc_flag="-Clinker-plugin-lto"
-#build:linux --@rules_rust//rust/settings:experimental_use_cc_common_link=True
-
-# This 'features' option comes from the `unix_cc_toolchain_config` in Bazel core and sets all the
-# right flags.
-#
-# See: <https://github.com/bazelbuild/bazel/blob/master/tools/cpp/unix_cc_toolchain_config.bzl>
-#build:release --features=thin_lto

doc/developer/bazel.md

@@ -42,11 +42,18 @@ allows Bazel to aggressively cache artifacts, which reduces build times.
   * [How Bazel Works](#how-bazel-works)
     * [`WORKSPACE`, `BUILD.bazel`, `*.bzl` files](#workspace-buildbazel-bzl-files)
     * [Generating `BUILD.bazel` files](#generating-buildbazel-files)
+      * [`cargo-gazelle`](#cargo-gazelle)
       * [Supported Configurations](#supported-configurations)
     * [Platforms](#platforms)
+      * [Custom Build Flags](#custom-build-flags)
     * [Toolchains](#toolchains)
-  * [`rules_rust`](#rules_rust)
+      * [System Roots](#system-roots)
+  * [Building Rust Code](#building-rust-code)
     * [`crates_repository`](#crates_repository)
+    * [Rust `-sys` crates](#rust--sys-crates)
+      * [Other C dependencies](#other-c-dependencies)
+    * [Protobuf Generation](#protobuf-generation)
+    * [Git Hash Versioning](#git-hash-versioning)
 
 # Getting Started
 
@@ -96,7 +103,8 @@ common --experimental_disk_cache_gc_max_age=7d
 ## Remote caching
 
 Bazel supports reading and writing artifacts to a [remote cache](https://bazel.build/remote/caching).
-We currently have two setup that are backed by S3 and running [`bazel-remote`](https://github.com/buchgr/bazel-remote).
+We currently have two setup in [`i2`](https://github.com/MaterializeInc/i2/blob/daacaca7e04e9c30e9914b5efa084047b3e58994/i2/apps/__init__.py#L126-L141)
+that are backed by S3 and running [`bazel-remote`](https://github.com/buchgr/bazel-remote).
 One is accessible by developers and used by PR builds in CI, we treat this as semi-poisoned. The
 other is only accessible by CI and used for builds from `main` and tagged builds.
 
@@ -135,15 +143,16 @@ it for other tools as well like `mzimage` and `mzcompose`! To enable Bazel speci
 flag like you would specify the `--dev` flag, e.g. `bin/mzcompose --bazel ...`.
 
 Otherwise Bazel can be used just like `cargo`, to build individual targets and run tests. We
-provide a thin wrapper around the `bazel` command in the form of `bin/bazel`. This sets up remote
-caching, and provides the `fmt` and `gen` subcommands. Otherwise it forwards all commands onto
-`bazel` itself.
+provide a thin wrapper around the `bazel` command in the form of [`bin/bazel`](../../misc/python/materialize/cli/bazel.py).
+This sets up remote caching, and provides the `fmt` and `gen` subcommands.
+Otherwise it forwards all commands onto `bazel` itself.
 
 ## Building a crate
 
-All Rust crates have a `BUILD.bazel` file that define different build targets for the crate. You
-don't have to write these files, they are automatically generated from the crate's `Cargo.toml`.
-For more details see the [Generating `BUILD.bazel` files](#generating-buildbazel-files) section.
+All Rust crates in our Cargo Workspace have a `BUILD.bazel` file that define different build
+targets for the crate. You don't have to write these files, they are automatically generated from
+the crate's `Cargo.toml`. For more details see the [Generating `BUILD.bazel` files](#generating-buildbazel-files)
+section.
 
 > **tl;dr** to build a crate run `bin/bazel build //src/<crate-name>` from the root of the repo.
 
@@ -264,7 +273,7 @@ There are three kinds of files in our Bazel setup:
 > **tl;dr** run `bin/bazel gen` from the root of the repository.
 
 Just like `Cargo.toml`, associated with every crate is a `BUILD.bazel` file that provides targets that
-Bazel can build. We auto-generate these files with a [`cargo-gazelle`](../bazel/cargo-gazelle/) which
+Bazel can build. We auto-generate these files with [`cargo-gazelle`](../../misc/bazel/cargo-gazelle) which
 developers can easily run via `bin/bazel gen`.
 
 There are times though when `Cargo.toml` doesn't provide all of the information required to build a
@@ -281,8 +290,22 @@ compile_data = ["path/to/my/file.txt"]
 This will add `"path/to/my/file.txt"` to the `compile_data` attribute on the
 resulting [`rust_library`](http://bazelbuild.github.io/rules_rust/defs.html#rust_library) Bazel target.
 
+### `cargo-gazelle`
+
+[`gazelle`](https://github.com/bazel-contrib/bazel-gazelle) is a semi-official `BUILD.bazel` file
+generator that supports Golang and protobuf. There exists a [`gazelle_rust`](https://github.com/Calsign/gazelle_rust)
+plugin, but it's not yet mature enough to fit our needs. Still, it's important for producivity that
+developers who don't want to interact with Bazel shouldn't have to, so generating a `BUILD.bazel`
+file from a `Cargo.toml` is quite important.
+
+Thus we decided to write our own generator, `cargo-gazelle`! It's not a plugin for the existing
+`gazelle` tool but theoretically could be. It's designed to be fully generic with very few (if any)
+Materialize specific configurations built in.
+
 ### Supported Configurations
 
+`cargo-gazelle` supports the following configuration in a `Cargo.toml` file.
+
 ```toml
 # Configuration for the crate as a whole.
 [package.metadata.cargo-gazelle]
@@ -403,12 +426,29 @@ serve:
    this is always the same as the "Host platform" since we don't utilize distributed builds.
 3. Target: the platform we are building for.
 
-The platforms that we build for are defined in [/platforms/BUILD.bazel](../bazel/platforms/BUILD.bazel).
+The platforms that we build for are defined in [`/platforms/BUILD.bazel`](../../misc/bazel/platforms/BUILD.bazel).
 
 A common way to configure a build based on platform is to use the
 [`select`](https://bazel.build/reference/be/functions#select) function. This allows you to return
 different values depending on the platform we're targetting.
 
+### Custom Build Flags
+
+Not necessarily related to platforms, but still defined in
+[`/platforms/BUILD.bazel`](../../misc/bazel/platforms/BUILD.bazel) are our custom build flags.
+Currently we have custom build settings for the following features:
+
+1. Sanitizers like AddressSanitizer (ASan).
+2. Cross language LTO
+
+While most build settings can get defined in the `.bazelrc` these features require slightly more
+complex configuration. For example, if we're building with a sanitizer we need to disable
+`jemalloc`, this is because sanitizers commonly have their own allocator. To do this we create a
+new build flag with the [`string_flag`](https://github.com/bazelbuild/bazel-skylib/blob/454b25912a8ddf3d90eb47f25260befd5ee274a8/docs/common_settings_doc.md)
+rule from the Bazel Skylib rule set and match on this using the [`config_setting`](https://bazel.build/docs/configurable-attributes)
+rule that is built in to Bazel. The [`config_setting`] is then what we can match on in our
+`BUILD.bazel` files with a `select({ ... })` function.
+
 ## Toolchains
 
 [Official Documentation](https://bazel.build/extending/toolchains)
@@ -418,32 +458,120 @@ specify a Rust toolchain every time you use the `rust_library` rule, you instead
 Rust toolchain that rules resolve during analysis.
 
 Toolchains are defined and registered in the [`WORKSPACE`](/WORKSPACE) file. We currently use
-Clang/LLVM to build C/C++ code, the version is defined by `LLVM_VERSION`, and we support both
-stable and nightly Rust, the versions defined by `RUST_VERSION` and `RUST_NIGHTLY_VERSION`
+Clang/LLVM to build C/C++ code (via the [`toolchains_llvm`](https://github.com/bazel-contrib/toolchains_llvm) ruleset)
+where the version is defined by the `LLVM_VERSION` constant. For Rust we support both stable and
+nightly, where the versions defined by the `RUST_VERSION` and `RUST_NIGHTLY_VERSION` constants
 respectively.
 
+Both [`toolchains_llvm`](https://github.com/bazel-contrib/toolchains_llvm/blob/0d3594c3edbe216e4734d52aa1e305049f877ea5/toolchain/osx_cc_wrapper.sh.tpl)
+and [`rules_rust`](https://github.com/bazelbuild/rules_rust/tree/e38fa8c2bc0990debceaf28daa4fcb2c57dcdc1c/util/process_wrapper) have "process wrappers".
+These are small wrappers around `clang` and `rustc` that are able to inspect the absolute path they
+are being invoked from. Bazel does not expose absolute paths _at all_ so these wrappers are how
+arguments like [`--remap-path-prefix`](https://doc.rust-lang.org/stable/rustc/command-line-arguments.html#--remap-path-prefix-remap-source-names-in-output)
+get set. These wrappers are helpful but can also cause issues like [`toolchains_llvm#421`](https://github.com/bazel-contrib/toolchains_llvm/issues/421).
+
 The upstream [LLVM toolchains](https://github.com/llvm/llvm-project/releases) are very large and
-built for bespoke CPU architectures. As such we build our own, see the
-[MaterializeInc/toolchains](https://github.com/MaterializeInc/toolchains) repo for more details.
+built for bespoke CPU architectures. While maybe not ideal, we build our own LLVM toolchains which
+live in the [MaterializeInc/toolchains](https://github.com/MaterializeInc/toolchains) repo. This
+ensures we're using the same version of `clang` across all architectures we support and greatly
+improves the speed of cold builds.
+
+> Note: The upstream LLVM toolchains are ~1 GiB and compressed with gzip, end-to-end they took
+about 3 minutes to download and setup. Our toolchains are ~80MiB and compressed with zstd which
+end-to-end take less than 30 seconds to download and setup.
+
+### System Roots
+
+Along with a C-toolchain we also provide a system root for our builds. A system root contains
+things like `libc`, `libm`, and `libpthread`, as well as their associated header files. Our system
+roots also live in the [MaterializeInc/toolchains](https://github.com/MaterializeInc/toolchains/blob/22e21deac9cec196c6adfcca811722882f92f941/sysroot/Dockerfile) repo.
 
-# [`rules_rust`](https://github.com/bazelbuild/rules_rust)
+# Building Rust Code
 
-For building Rust code we use `rules_rust`. It's primary component is the
-[`crates_repository`](http://bazelbuild.github.io/rules_rust/crate_universe.html#crates_repository) rule.
+For building Rust code we use [`rules_rust`](https://github.com/bazelbuild/rules_rust). It's
+primary component is the [`crates_repository`](http://bazelbuild.github.io/rules_rust/crate_universe.html#crates_repository)
+rule.
 
 ## `crates_repository`
 
 Normally when building a Rust library you define external dependencies in a `Cargo.toml`, and
-`cargo` handles fetching the relevant crates, generally from `crates.io`. The [`crates_repository`]
+`cargo` handles fetching the relevant crates, generally from `crates.io`. The [`crates_repository`](https://bazelbuild.github.io/rules_rust/crate_universe_workspace.html#crates_repository)
 rule does the same thing, we define a set of manifests (`Cargo.toml` files), it will analyze them
 and create a Bazel [repository](https://bazel.build/external/overview#repository) containing all of
 the necessary external dependencies.
 
-Then to build our crates, e.g. [`mz-adapter`](/src/adapter/), we use the handy
+Then to build our crates, e.g. [`mz-adapter`](../../src/adapter/), we use the handy
 [`all_crate_deps`](http://bazelbuild.github.io/rules_rust/crate_universe.html#all_crate_deps)
 macro. When using this macro in a `BUILD.bazel` file, it determines which package we're in (e.g.
 `mz-adapter`) and expands to all of the necessary external dependencies. Unfortunately it does not
 include dependencies from within our own workspace, so we still need to do a bit of manual work
 of specifying dependencies when writing our `BUILD.bazel` files.
 
-In the [`WORKSPACE`](/WORKSPACE) file we define a "root" `crates_repository` named `crates_io`.
+In the [`WORKSPACE`](../../WORKSPACE) file we define a "root" `crates_repository` named `crates_io`.
+
+## Rust `-sys` crates
+
+There are some Rust crates that are wrappers around C libraries, like
+[`decnumber-sys`](https://crates.io/crates/decnumber-sys) is a wrapper around
+[`libdecnumber`](https://speleotrove.com/decimal/). `cargo-gazelle` will generate a Bazel target
+for the crate's build script, but it's likely this build script will fail because it can't find
+tools like `cmake`, our system root, or implicitly depends on some other C library.
+
+The general approach we've used to get these crates to build is to duplicate the logic from the
+`-sys` crate's `build.rs` script into a Bazel target. See
+[bazel/c_deps/rust-sys](../../misc/bazel/c_deps/rust-sys) for some examples. Once you write a
+`BUILD.bazel` file for the C dependency we add a `crate.annotation` in our [`WORKSPACE`](https://github.com/MaterializeInc/materialize/blob/5f2f45a162c44e4c6a03ba017f8b7d1d00c3775b/WORKSPACE#L464-L469)
+file that appends your newly written `BUILD.bazel` file to the one generated for the Rust crate.
+
+Duplicating logic is never great, but having Bazel explicitly build these C dependencies provides
+better caching and more control over the process which unlocks features like cross language LTO.
+
+### Other C dependencies
+
+There are a few C dependencies which are used both by a Rust `-sys` crate and another C dependency.
+For example `zstd` is used by both the `zstd-sys` Rust crate and the `rocksdb` C library. For these
+cases instead of depending on the version included via the Rust `-sys` crate, we "manually" include
+them by downloading the source files as an [`http_archive`](https://bazel.build/rules/lib/repo/http).
+All cases of external C dependencies live in [`bazel/c_deps/repositories.bzl`](../../misc/bazel/c_deps/repositories.bzl).
+
+## Protobuf Generation
+
+Nearly all of our Rust build scripts do a single thing, and that's generate Rust bindings to
+protobuf definitions. `rules_rust` includes [rules for generating protobuf bindings](https://bazelbuild.github.io/rules_rust/rust_prost.html)
+when using Prost and Tonic, but they don't interact with Cargo Build Scripts very well. Instead we
+added a new crate called [`build-tools`](../../src/build-tools) whose purpose is to abstract over
+whatever build system you're using and provide the tool a build script might need, like `protoc`.
+
+For Bazel we provide the necessary tools via ["runfiles"](https://bazel.build/rules/lib/builtins/runfiles), which are defined in the [`data` field](https://github.com/MaterializeInc/materialize/blob/5f2f45a162c44e4c6a03ba017f8b7d1d00c3775b/src/build-tools/BUILD.bazel#L30-L33)
+of the `rust_library` target. Bazel "runfiles" are a set of files that are provided at runtime
+execution. So in your build script to get the current path of the `protoc` executable you would
+call `mz_build_tools::protoc` ([example](https://github.com/MaterializeInc/materialize/blob/5f2f45a162c44e4c6a03ba017f8b7d1d00c3775b/src/persist/build.rs#L14))
+which returns a [different path depending on the build system](https://github.com/MaterializeInc/materialize/blob/5f2f45a162c44e4c6a03ba017f8b7d1d00c3775b/src/build-tools/src/lib.rs#L38-L64)
+currently being used.
+
+## Git Hash Versioning
+
+Development builds of Materialize include the current git hash in their version number. The sandbox
+that Bazel creates when building a Rust library does not include any git info, so attempts to get
+the current hash will fail.
+
+But! Bazel has a concept of "stamping" builds which allows you to provide local system information
+as part of the build process, this information is known as the [workspace status](https://bazel.build/docs/user-manual#workspace-status).
+Generating the workspace status and providing it to Rust libraries requires a few steps, all of
+which are described in the [`bazel/build-info/BUILD.bazel`](https://github.com/MaterializeInc/materialize/blob/5f2f45a162c44e4c6a03ba017f8b7d1d00c3775b/misc/bazel/build-info/BUILD.bazel#L12-L29) file.
+
+Unfortunately this isn't the whole story though. It turns out workspace status and stamping builds
+causes poor remote cache performance. On a new build Bazel will regenerate the `volatile-status.txt`
+file used in workspace stamping which causes any stamped libraries to not be fetched from the
+remote cache, see [`bazelbuild#10075`](https://github.com/bazelbuild/bazel/issues/10075). For us
+this caused a pretty serious regression in build times so we came up with a workaround:
+
+1. When [building in release mode but not a tagged build](https://github.com/MaterializeInc/materialize/blob/e427108049dd97f55b8b6d634ae01b025b8520b5/misc/python/materialize/mzbuild.py#L156-L163),
+   (e.g. a PR) `mzbuild.py` will [write out the current git hash to a temporary file](https://github.com/MaterializeInc/materialize/blob/e427108049dd97f55b8b6d634ae01b025b8520b5/misc/python/materialize/bazel.py#L39-L64).
+2. Our `build-info` Rust crate knows to [read from this temporary file](https://github.com/MaterializeInc/materialize/blob/main/src/build-info/src/lib.rs#L123-L141)
+   in a non-hermetic/side-channel way to get the git hash into the current build without
+   invalidating the remote cache.
+
+While definitely hacky, our side-channel for the git hash does provide a substantial improvement in
+build times, while providing similar guarantees to the Cargo build with respect to when the hash
+gets re-computed.