"Executing WebAssembly in repository rules"

[lberki]

This is the discussion thread for @jmillikin 's proposal Executing WebAssembly in repository rules.

[lberki]

@meteorcloudy @Wyverald @meisterT please weigh in!

[meisterT]

Re: "Download a pre-compiled WebAssembly runtime"

Can we use this approach as a stepping stone to test out whether and how it is adopted? If it is widely successful, we can smoothen the integration in Bazel at a later point. Seems to me also to be an integral part of supporting older Bazel versions anyway.

You state as one of the downsides selecting/downloading precompiled binaries. Is there some shared/common infra that we could provide to ruleset authors so not everyone has to do it by themselves?

[meteorcloudy]

Is there some shared/common infra that we could provide to ruleset authors so not everyone has to do it by themselves?

Exactly, couldn't we encapsulate the whole logic in rules_wasm and just expose a execute_wasm API from it?

[meteorcloudy]

Wait, there is already https://github.com/vimana-cloud/rules_wasm

[jmillikin]

Re: "Download a pre-compiled WebAssembly runtime"

Can we use this approach as a stepping stone to test out whether and how it is adopted?

I wrote this in the doc, but I don't think this would be a useful metric. There's absolutely no value in downloading a pre-compiled native binary to execute WebAssembly that could have just been compiled to its own native binary with the same platform coverage, so I would expect the adoption of such a shim to be low regardless of whether supporting WebAssembly in Bazel would be popular.

The reason to use WebAssembly at all is to avoid the need for pre-compiled native binaries. If that reason is removed, then all you're left with is a slightly slower execution strategy.

If it is widely successful, we can smoothen the integration in Bazel at a later point. Seems to me also to be an integral part of supporting older Bazel versions anyway.

I think the repeated use of pre-compiled native binaries shows that there is broad demand for some sort of non-Starlark repository rule extension mechanism that's able to leverage third-party libraries from outside the Bazel ecosystem.

Supporting older Bazel versions doesn't seem that important, given that anyone with an existing ruleset based on native binaries can just continue to do whatever they're currently doing. The WebAssembly support can be used as a fallback for otherwise unsupported platforms until that ruleset's minimum Bazel version is >= v9 (or v10, or whatever).

If there is a ruleset developer out there who wants to switch to a WebAssembly-only approach and also is willing to maintain some sort of wazero-based shim, then there's nothing stopping them from doing that right now. There's no technical blockers, Bazel has supported repository_ctx.execute() since the pre-v1.0 days.

Here's a quick-n-dirty wazero-shim.go that they can use as a starting point.

`wazero-shim.go` (~70 lines)

// SPDX-License-Identifier: 0BSD
package main

import (
	"context"
	"fmt"
	"os"

	"github.com/tetratelabs/wazero"
)

func execWasm(ctx context.Context) error {
	bytecodePath := os.Args[1]
	wasmFnName := os.Args[2]
	inputPath := os.Args[3]
	outputPath := os.Args[4]

	bytecode, err := os.ReadFile(bytecodePath)
	if err != nil { return err }
	input, err := os.ReadFile(inputPath)
	if err != nil { return err }

	cfg := wazero.NewRuntimeConfigInterpreter()
	runtime := wazero.NewRuntimeWithConfig(ctx, cfg)
	defer runtime.Close(ctx)

	module, err := runtime.CompileModule(ctx, bytecode)
	if err != nil { return err }
	moduleCfg := wazero.NewModuleConfig().WithStartFunctions("_initialize")
	instance, err := runtime.InstantiateModule(ctx, module, moduleCfg)
	if err != nil { return err }

	mem := instance.Memory()
	allocFn := instance.ExportedFunction("bazel_wasm_allocate")
	function := instance.ExportedFunction(wasmFnName)

	inputLen := uint64(len(input))
	allocResult, err := allocFn.Call(ctx, inputLen)
	if err != nil { return err }
	inputPtr := allocResult[0]
	mem.Write(uint32(inputPtr), input)

	allocResult, err = allocFn.Call(ctx, 8)
	if err != nil { return err }
	paramsPtr := allocResult[0]
	outputPtrPtr := paramsPtr
	outputLenPtr := paramsPtr + 4
	mem.WriteUint32Le(uint32(outputPtrPtr), 0)
	mem.WriteUint32Le(uint32(outputLenPtr), 0)

	execResult, err := function.Call(ctx, inputPtr, inputLen, outputPtrPtr, outputLenPtr)
	if err != nil { return err }
	rc := int32(execResult[0])

	outputPtr, _ := mem.ReadUint32Le(uint32(outputPtrPtr))
	outputLen, _ := mem.ReadUint32Le(uint32(outputLenPtr))
	output, _:= mem.Read(outputPtr, outputLen)
	if err := os.WriteFile(outputPath, output, 0o600); err != nil {
		return err
	}
	os.Exit(int(rc))
	return nil
}

func main() {
	if err := execWasm(context.Background()); err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
}

You state as one of the downsides selecting/downloading precompiled binaries. Is there some shared/common infra that we could provide to ruleset authors so not everyone has to do it by themselves?

Bazel provides repository_ctx.os.{arch,name} for querying the host platform, and it's also possible to execute uname -ms in cases where the Java OS properties are unclear for whatever reason.

See rules_rust for an example of how this is used in practice:

• get_host_triple() returns a normalized version of repository_ctx.os attributes.
• _get_generator() downloads a pre-compiled cargo-bazel binary that matches the host platform.
• https://github.com/bazelbuild/rules_rust/releases/tag/0.59.0 has 8 cargo-bazel binaries available for various {arch}-{os}-{libc} combinations.

Wait, there is already https://github.com/vimana-cloud/rules_wasm

That repository appears to be about compiling to WASI modules during the build stage. I might just be overlooking it, but I don't see any obvious repository rule functionality there.

[lberki]

I'm not too enthusiastic about this idea for two reasons:

1. This would add another, very special interface to Bazel which has so far done a reasonably good job at making all of its interfaces Starlark.
2. This would allow people to run arbitrary code in the address space of Bazel. This creates security issues in case Bazel can't trust everyone who writes BUILD/.bzl files and also makes it possible to use an arbitrary amount of RAM and CPU without a clear indication as to where that use is coming from.

In addition to these, this would not provide a lot of additional value over downloading a pre-compiled WASM runtime. Yes, I do realize that @jmillikin is explicitly saying above that a pre-compiled WASM runtime doesn't bring too much value, but I don't buy that argument: it hinges on pre-compiled binaries not being good enough, which would require that platforms for which one can't provide one see a lot of use. Bazel provides binaries for six combinations of ARM64/x64 and Linux/Mac OS/Windows, which I hope covers most of the use, otherwise we should be providing more pre-compiled binaries! (I do no have the telemetry to say that definitively, though)

[lberki]

Do keep in mind that this is not just a tradeoff between portability and complexity. We'd also invite user code to run within the address space of Bazel, which we so far haven't done except through Starlark and which seems like a big cost to cater for what are admittedly niche platforms -- if they were not niche, surely we'd produce prebuilt Bazel binaries for them?

[jmillikin]

@Wyverald

Thank you very much for your considerate and thoughtful response! I can see where you're coming from regarding the different interpretations of the proposal's value (portability of arch-independent bytecode vs reducing boilerplate for existing native binaries), and I'll see if I can update the proposal to explain more about why I think portability is the more compelling benefit in this case.

In summary (I'll try to expand in the proposal), the building of native binaries and selection among them in a repository rule is pretty simple if the available platforms are limited. For example, a selection of native binaries for {aarch64,x86_64}-{windows,linux,macos} is easy for maintainers to manage and probably covers over 90% of Bazel users. Assuming the rules_wasm_repotool maintainers support the same platforms that Bazel provides official binary builds for, the amount of code within rules_wasm_repotool would actually be very small. For an example, @rules_rust//rust/platform:triple.bzl%get_host_triple is ~40 lines of Starlark.

Therefore what rules_wasm_repotool would provide is less about the Starlark logic and more about pre-compiled binaries for lots and lots of cpu/os combinations. That's not worthless, but I think it's difficult to make a case for its value being worth the significant maintenance burden. Also, from the rules author's perspective, it adds maintenance burden when users request updates to the rules_wasm_repotool dependency to pick up new os/arch combinations.

@lberki

I think my first comment reply covered your concerns regarding executing user-provided WebAssembly within the Bazel address space. Assuming the presence of CPU and/or RAM limits on the WebAssembly VM, I don't understand why user-provided WebAssembly would be different from user-provided Starlark.

Would the proposal's variant of having Bazel fork() before running WebAssembly resolve the address space issue?

Alternatively, could you share more detail about the downsides you foresee with WebAssembly in the same address space? For example, is it something related to Java GC pressure? Interference with memory/CPU performance profiling?

Regarding whether Bazel rulesets should support platforms that do not receive an official binary release from the Bazel team, I think it would be difficult for you and me to reach consensus on that point, so noting the disagreement and carrying on might be best.

[lberki]

@jmillikin am I holding this thread wrong? I don't see your "first comment reply" you mention above (I do seem to remember this argument from before, so I probably am holding this thread wrong...)

It seems that the main argument against rules_wasm_repotool is that it doesn't provide a way to support the "long tail" platforms which require their own pre-compiled binary, which the owners of rules_wasm_repotool are not willing to provide. I think that's an argument that has some merit, but how much merit it has exactly depends on how much use the "long tail" platform see and I don't have data on that. Maybe @meteorcloudy does?

The part I don't understand about the portability argument is that it looks like two conditions that must be fulfilled for the Chicory approach to be easier on any given platform than the rules_wasm_repotool approach: That platform must not have a pre-built WASM runtime in rules_wasm_repotool, but it must have a Bazel binary. Does this happen frequently? Again, I don't know, but naively, one would think that if one goes through the trouble of building Bazel for a platform, they could just as well do the same with a WebAssembly runtime and thus a priori I'd think that this is a rare occurrence.

re: allowing running non-Starlark user code in the address space of Bazel: all of the above :) Forking would indeed make using Chicory much more palatable. I'd much enjoy not having to field bugs that report Bazel using too much CPU or RAM or deadlocking, only to find out that it was some WASM code that is the culprit, but which is not apparent without digging deep.

From a cursory glance at the documentation of Chicory, it looks like it does support hard RAM limits (good) and relies on Thread.interrupt() to enforce CPU limits (tolerable, but not good -- this means that whether your function runs depends on the CPU it's running on).

Regarding whether Bazel rulesets should support platforms that do not receive an official binary release from the Bazel team, I think it would be difficult for you and me to reach consensus on that point

I don't have a firm opinion on the matter. Did I say something to the contrary above?

[meisterT]

@lberki github displays "show N previous replies" above if there are too many comments - @jmillikin is referring to this comment: #25537 (reply in thread)

[lberki]

@meisterT thanks, that's embarrassing :)

[fmeum]

@jmillikin Have you done any benchmarks for Chicory? For example, how long does it take to parse and generate the BUILD files for some repo rule that currently relies on a native binary?

[jmillikin]

I haven't done benchmarks myself, all the things I use repository rule helper tools for are small enough that they finish ~instantly in any implementation.

• In my experience the WebAssembly runtimes that do some sort of JIT are reasonably fast (between 50% and 90% performance vs native code compiled with -O2), and the pure interpreters are quite slow (something like 5% to 10% of native).
• Extracting Go imports using the Go parser's ImportsOnly mode is cheap enough that even an interpreter is fine, but running gofmt.wasm on a 10,000 LoC input file has a human-visible delay before the command finishes.

The Chicory team has published benchmarks of their v1.0.0 against GraalWasm, which compiles to native code: https://chicory.dev/blog/chicory-1.0.0/#the-race-day

The MVP implementation in my branch uses Chicory's interpreter because I was more interested in demonstrating the Starlark API part of the proposal, and didn't want to get bogged down in learning about Java dynamic bytecode loading. If the proposal is adopted it might be worth having an --experimental_wasm_jit flag that enables Chicory's JIT. If the JIT works without problems then it could become the default.

[sluongng]

I wonder if we can solve the problem using a different solution. Recently, there have been a few talks about "Build rules generating BUILD files", which may let us shift the BUILD file generation into Bazel action graph. I know that any possible solution is quite distant away, but if that feature is available, then the "BUILD gen" action will be subjected to w/e tool and sandbox option configured by user code, bypassing the need for wasm-support in rctx?

[Wyverald]

A few of us met in a VC to discuss this. We've decided to approve this proposal.

• Running WASM in repo rules allows rules authors to take advantage of helper libraries in their favorite (WASM-targeting) languages and still enjoy portability benefits.
• Supporting niche platforms is not worth much of our brain time, but this proposal also doesn't add much downside, since Chicory is single-threaded, can be RAM bound, and doesn't add much to the binary size.
• We should add an API with a timeout (potentially configurable), gated behind an experimental flag to begin with.

[thesayyn]

I believe this makes a great sense and will ease the pain of maintainers in a lot of areas and I do agree with the portability argument.

Here is some areas where i think a native support will greatly help.

• Gazelle being built from source, now instead can be prebuilt into wasm and executed in the repository context. Gazelle/Go does this in order to generate build files for external deps.
• Lockfile parsers / DAG-AD resolvers
• TOML,YAML, BYOL parsers.