[NativeAOT-LLVM] Add a property to build Wasm core modules for library project types

[yowl]

This PR adds a property WasmCoreModule that builds a Wasm core module as opposed to a component module. This is useful when targeting hosts that do not support components, e.g. extism. While this works for the simple cases, I suspect there is more to it. Extism has their own API for modules and doesn't seem like there is much appetite to change to the component module.

[SingleAccretion]

This would be a configuration that is expensive to support with our current infrastructure. There are two problems:

1. We build runtime objects (libPortableRuntime.a / libSystem.Native.a and others included ICU, which is built in dotnet/icu) with a libc that targets WASIp2. The compatibility story between WASIp1 and WASIp2 when using wasi-libc is complicated; I think it would be fair to summarize it as "shaky at best" and certainly "not guaranteed". In other words, to making this work robustly would require a separate build of all runtime objects that'd target WASIp1. Naturally in this mode some functionality would not be available at all (e. g. sockets).
   • Another option here is compiling the runtime object on the user's machine, there is some precedent for that strategy.
2. We pierce the wasi-libc abstraction and import WASIp2 directly in some libraries (e. g. http).

There was some discussion at some point about a "module-only" version of the WASIp2 targets that'd solve the component-model-less runtime problem (assuming they'd implement [a subset of] WASIp2), but it hasn't really gone anywhere yet.

Ref: WebAssembly/wasi-libc#447.

[yowl]

Struggling a bit with this arm laptop, sorry about not testing this.

[jkotas]

If the typo is arm instead of amd next time, the arm laptop will forgive you

[yowl]

I'm missing something, this is a Snapdragon Surface 7

[yowl]

Are the wasi people saying they can't produce core wasm modules any more, seems like a strange restriction?

[SingleAccretion]

Are the wasi people saying they can't produce core wasm modules any more, seems like a strange restriction?

For WASIp2, yeah, it "needs work", mainly around making wasi-libc use the p2 imports directly instead of the current libc -> p1 -> p1->p2 adapter -> p2 setup. See also WebAssembly/wasi-sdk#426.

Furthermore, it would need work in the hosts (like extism) to actually implement the p2 imports.

[yowl]

At what point would you expect it to break, as simple things seem to work?

[SingleAccretion]

At what point would you expect it to break, as simple things seem to work.

It is mainly a question of wasi-libc evolution. As more ABI breaks (between p1 and p2 implementations) are introduced and more p2 imports used directly, the more stuff will break. As I said, the only way to avoid this is to either ship a p1-targeting set of objects, or let the user build them themselves (i. e. ship source).

[yowl]

How about we let users specify WasmLinkerPath and they can take the risks themselves. I.e. users replace WasmLinkerPath with something that changes link.rsp as they wish before calling wasm-ld. The triple they can already override without this PR. Can link.exe be overriden in the non-wasm case?

[SingleAccretion]

I don't think we need new extension points for "hacky" solution. If you already have the libraries built against WASIp1 (i. e. a custom dotnet/runtime/NativeAOT-LLVM) build, you can override IlcFrameworkNativePath and IlcSdkPath and the build will link in your overrides.

I think the tricky part of the above is the "custom runtimelab build" stuff, not the (overall minor) tweaks to the targets required to enable it.

In truth I would rather like for this option to use p1 to exist. Just really don't know (currently at least) how to fit it into the build/packaging/testing system so that it is both not intrusive and actually known to work.

[yowl]

I'm not sure in my mind how much this should depend on the version of wasi. I'd also like it to work for zero sharp scenarios with no libc.

[SingleAccretion]

I'd also like it to work for zero sharp scenarios with no libc.

Those already work, do they not? Since they control all native code compilation/linking.

[yowl]

I'd also like it to work for zero sharp scenarios with no libc.

Those already work, do they not? Since they control all native code compilation/linking.

Yes, I suppose so. I wonder what Rust does to support extism and p2.

[SingleAccretion]

Yes, I suppose so. I wonder what Rust does to support extism and p2.

I believe it uses the "compile all runtime objects separately" solution. In Rust it's easier since they have separate targets for p1 and p2, while we don't have separate RIDs (there is only wasi-wasm).

[yowl]

We allow the user to change the triple with IlcLlvmTarget, it would seem to be coherent to remove Wl,--component-type if its not wasm32-unknown-wasip2 and as no other triple will do components (wasip3 will). Would that be ok?

(The c SDK for extism uses wasm32-unknown-unknown with -nostdlib)

Edit, extism doesn't use wasi at all, we can't wait for them to upgrade, as there is nothing for them to upgrade. I think a NAOT-LLVM wasip2 linked component is fine as far as they are concerned.

[SingleAccretion]

We allow the user to change the triple with IlcLlvmTarget

(Nit: this is an internal property, it can only be changed insofar as all properties in MSBuild can be changed. It wasn't meant to be a user-visible knob.)

Edit, extism doesn't use wasi at all, we can't wait for them to upgrade, as there is nothing for them to upgrade. I think a NAOT-LLVM wasip2 linked component is fine as far as they are concerned.

Hmm, not sure I understand. If they don't use WASI, how can you use any object from this repo at all (that expect WASI imports to be present) as-is, without going zerosharp? Put another way: what are we missing to enable the extism scenario?

[yowl]

Hmm, not sure I understand. If they don't use WASI, how can you use any object from this repo at all (that expect WASI imports to be present) as-is, without going zerosharp? Put another way: what are we missing to enable the extism scenario?

You can't. Extism provides some IO things, like HTTP requests and logging through its own exports which are made available via it's SDK. That doesn't mean that you can't do useful things as a plugin author in C# though.

[SingleAccretion]

Extism provides some IO things, like HTTP requests and logging through its own exports which are made available via it's SDK. That doesn't mean that you can't do useful things as a plugin author in C# though.

So you're saying we don't have anything we should/can fix/improve in-tree to enable the extism scenario?

That puts the conversation back into other more traditional wasip1 module use cases. I have actually come across this thing recently: https://github.com/cpetig/wasmtime-adapter (linking in the adapter statically instead of as a component). Maybe we could create a sample using this and then fix whatever needs to fixing to have it work?