[NativeAOT-LLVM] Wasm managed threads - build new nupkg #3060

yowl · 2025-03-30T18:59:08Z

This PR provides a base for working on multithread support, as and when we have Wasm runtimes that support threads. With the browser this could be done now using Web Workers, the same as mono, for WASI with wasmtime we will need to wait for at least preview3 v2, possibly later.

No actual work for the dotnet runtime functionality that we will need, e.g. GC polling is included, this PR just creates CI jobs, runs the tests that passed, and creates 2 new packages for testing.

jkotas · 2025-05-26T00:56:34Z

...stem.Runtime.InteropServices.JavaScript/src/System.Runtime.InteropServices.JavaScript.csproj

@@ -76,6 +76,11 @@
    <Compile Include="System\Runtime\InteropServices\JavaScript\JSWebWorker.cs" />
    <Compile Include="System\Runtime\InteropServices\JavaScript\JSSynchronizationContext.cs" />
    <Compile Include="System\Runtime\InteropServices\JavaScript\JSAsyncTaskScheduler.cs" />
+    <!-- Can't build System.Threading.Channels.csproj without these -->


Why is that? This workaround does not look right.

Thanks for looking, it was as though even though these transitive dependencies were built, they weren't found for this project. However it doesn't seem consistent and I removed this change now.

yowl · 2025-07-05T18:39:36Z

Cc @dotnet/nativeaot-llvm

SingleAccretion

Just a bit of a skim so far, will continue tomorrow.

SingleAccretion · 2025-07-05T20:08:42Z

eng/pipelines/common/platform-matrix.yml

@@ -500,6 +500,44 @@ jobs:
        buildConfig: ${{ parameters.buildConfig }}
        ${{ insert }}: ${{ parameters.jobParameters }}

+- ${{ if containsValue(parameters.platforms, 'wasi_multithread_wasm_win') }}:


I don't think we need these new platforms; we should be able to use the existing ones like here: https://github.com/dotnet/runtime/blob/9e5e6aa7bc36aeb2a154709a9d1192030c30a2ef/eng/pipelines/common/templates/wasm-build-only.yml#L17-L36, utilizing the nameSuffix parameter to global-build-job.yml.

SingleAccretion · 2025-07-05T20:10:47Z

docs/workflow/building/coreclr/nativeaot.md

+
+This is a work in progress and far from functional.  Currentlty there exists just enough infrastucture to build packages for mutlithreaded runtime and libs, but they are not functional in the sense that they support multithreaded programs yet.  To build the WASI packages:
+```
+build clr.aot+libs+nativeaot.packages -c Debug -a wasm -os wasi -cmakeargs -DCLR_CMAKE_TARGET_OS_SUBGROUP=multithread '/p:WasmEnableThreads=true'


We should funnel /p:WasmEnableThreads down to cmake/clang automatically. E. g. the libs native build sets CMAKE_USE_PTHREADS, we can extended that pattern to the CoreCLR's runtime.proj.

SingleAccretion · 2025-07-05T20:12:25Z

eng/pipelines/runtimelab/runtimelab-post-build-steps.yml

@@ -31,8 +31,17 @@ steps:
        displayName: Set up ILC emulation environment

    - ${{ if eq(parameters.archType, 'wasm') }}:
-      - script: $(Build.SourcesDirectory)/src/tests/build$(scriptExt) nativeaot $(buildConfigUpper) ${{ parameters.osGroup }} $(crossArg) $(_officialBuildParameter) ci tree nativeaot /p:LibrariesConfiguration=${{ parameters.librariesConfiguration }}
-        displayName: Build runtime tests
+      - ${{ if eq(parameters.platform, 'browser_multithread_wasm_win') }}:


It would be good to avoid this duplication. We can add a parameter to these post-build steps like wasmEnableThreadsArg: /p:WasmEnableThreads=true and use it here (and below to excluded the libs tests).

SingleAccretion · 2025-07-05T20:13:15Z

src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.targets

@@ -438,6 +438,7 @@ The .NET Foundation licenses this file to you under the MIT license.
      <CompileWasmArgs Condition="'$(NativeDebugSymbols)' == 'true'">$(CompileWasmArgs) -g3</CompileWasmArgs>
      <CompileWasmArgs Condition="'$(WasmEnableNonTrappingFloatToIntConversions)' == 'true'">$(CompileWasmArgs) -mnontrapping-fptoint</CompileWasmArgs>
      <CompileWasmArgs Condition="'$(IlcLlvmExceptionHandlingModel)' == 'wasm'">$(CompileWasmArgs) -fwasm-exceptions</CompileWasmArgs>
+      <CompileWasmArgs Condition="'$(WasmEnableThreads)' == 'true'">$(CompileWasmArgs) -pthread -matomics -mbulk-memory</CompileWasmArgs>


You also need to modify the IlcLlvmTarget to include -threads I think.

I didn't think there were any triples for threads, at least not yet.

There is wasm32-wasi-threads, but looks like we don't need anything for Emscripten indeed and -pthread is enough.

SingleAccretion · 2025-07-05T20:13:58Z

src/coreclr/nativeaot/Runtime/unix/PalRedhawkUnix.cpp

+#elif defined(HOST_WASM) && defined(FEATURE_WASM_MANAGED_THREADS)
+        PalGetMaximumStackBounds_MultiThreadedWasm(&pStackLowOut, &pStackHighOut);


This should use the pthread_attr_init code from below instead of a bespoke version.

Thanks, have updated this and removed PalGetMaximumStackBounds_MultiThreadedWasm with the logic move to pthread_attr_init and pthread_attr_getstack

SingleAccretion · 2025-07-05T20:15:34Z

src/native/external/zlib-ng.cmake

+      add_compile_options(-D_WASI_EMULATED_PTHREAD)
+      add_linker_flag(-Wl,-lwasi-emulated-pthread)


Why didn't we need this before?

Seems we don't need it after all. Removed.

yowl · 2025-07-11T01:23:06Z

What is it about PackagingTests.csproj that changes the way the target runtime package name is generated?

SingleAccretion · 2025-07-11T10:35:25Z

What is it about PackagingTests.csproj that changes the way the target runtime package name is generated?

The packaging test implementation project (HelloWasm\PackagingTests\PackagingTestProject.csproj) receives the things it needs from the parent project (HelloWasm\PackagingTests.csproj) via the Test_ properties. The target package name, however, is actually not decided by these properties but by the target RID via nuget runtime.json magic. Since we don't have a target RID for multi-threaded WASM (a bit of a design flaw in the current scheme, IMO, but that ship has sailed long ago), we're in a bit of a pickle. Maybe disable the packaging test for MT and leave solving this problem for later?

…r_getstack

SingleAccretion

Larger overall point on testing.

We currently don't have even the basic support for e. g. correct memory orderings, so stuff will fail due to race conditions randomly in CI. Should we disable all tests (and only do build) for now, before we know the configuration is at least not 100% known to be flaky? Alternatively, only enable like one test (e. g. HelloWasm)?

SingleAccretion · 2025-07-15T19:11:29Z

eng/native/gen-buildsys.cmd

+        if defined __CMakeArgs (
+            if not "!__CMakeArgs:multithread=!" == "!__CMakeArgs!" (
+                set __ExtraCmakeParams=%__ExtraCmakeParams% -D_WASI_EMULATED_PTHREAD
+            )
+        )


eng/pipelines/runtimelab/runtimelab-post-build-steps.yml

SingleAccretion · 2025-07-15T19:12:32Z

eng/pipelines/runtimelab/runtimelab-post-build-steps.yml

@@ -17,21 +18,25 @@ steps:
      displayName: Build the ILC and RyuJit cross-compilers

    # Build target packages (note: target libs already built).
-    - script: $(Build.SourcesDirectory)/build$(scriptExt) nativeaot.packages -os ${{ parameters.osGroup }} -a wasm -c $(buildConfigUpper) $(_officialBuildParameter) -ci
-      displayName: Build target packages
+    - ${{ if ne(parameters.nameSuffix, 'Multithreaded') }}:


Can use wasmEnableThreadsArg here.

SingleAccretion · 2025-07-15T19:13:15Z

eng/pipelines/runtimelab/runtimelab-post-build-steps.yml


    # Build host packages.
    - script: $(Build.SourcesDirectory)/build$(scriptExt) libs+nativeaot.packages -a $(hostedTargetArch) -c $(buildConfigUpper) -cross $(_officialBuildParameter) -ci
      displayName: Build host packages

  # Build coreclr native test output outside of official build
  - ${{ if ne(parameters.isOfficialBuild, true) }}:
-    - ${{ if and(eq(parameters.archType, 'wasm'), ne(parameters.nameSuffix, '')) }}:
+    - ${{ if and(eq(parameters.archType, 'wasm'), and(ne(parameters.nameSuffix, ''), ne(parameters.nameSuffix, 'Multithreaded'))) }}:


Better to use wasmEnableThreadsArg != '' here, the existing nameSuffix check is already a bit of a hack.

SingleAccretion · 2025-07-15T19:13:31Z

eng/pipelines/runtimelab/runtimelab-post-build-steps.yml

@@ -49,7 +54,7 @@ steps:
        displayName: Run runtime tests

    # Don't compile/run the libraries tests with emulated ILC to save CI time/resources.
-    - ${{ if and(eq(parameters.archType, 'wasm'), eq(parameters.nameSuffix, '')) }}:
+    - ${{ if and(and(eq(parameters.archType, 'wasm'), eq(parameters.nameSuffix, '')), not(eq(parameters.nameSuffix, 'Multithreaded'))) }}:


Ditto with wasmEnableThreadsArg.

SingleAccretion · 2025-07-15T19:20:04Z

eng/pipelines/runtimelab.yml

+            osSubgroup: multithread
+            platforms:
+            - browser_wasm_win
+            - wasi_wasm_win


How is this going to work with WASI, considering we don't have a -threads triple for WASIp2 yet? I would focus on making the fundamentals work on browser for now, we can enable WASI later when the libc support comes along.

SingleAccretion · 2025-07-15T19:22:25Z

src/coreclr/nativeaot/Runtime/wasm/PalRedhawkWasm.cpp

+#ifdef FEATURE_WASM_MANAGED_THREADS
+int __cxa_thread_atexit(void (*func)(), void *obj, void *dso_symbol)
+{
+    return 0;
+}
+
+//
+// Note that we return the native stack bounds here, not shadow stack ones. Currently this functionality is mainly
+// used for RuntimeHelpers.TryEnsureSufficientExecutionStack, and we do use the native stack in codegen, so this
+// is an acceptable approximation.
+//
+extern "C" unsigned char __stack_low;
+extern "C" unsigned char __stack_high;
+


There is a lot of code copied here and it's not clear what part of it is needed and what part is there because WASI doesn't actually -pthread support yet.

SingleAccretion · 2025-07-15T19:23:49Z

src/coreclr/nativeaot/System.Private.CoreLib/src/System.Private.CoreLib.csproj

@@ -35,6 +35,11 @@
  <PropertyGroup Condition="'$(TargetsSingleThreadedWasm)' != 'true'">
    <FeaturePortableThreadPool>true</FeaturePortableThreadPool>
    <FeaturePortableTimer>true</FeaturePortableTimer>
+    <!-- LLVM-TODO: Remove FeatureWasmManagedThreads in favor of WasmEnableThreads. -->
+    <FeatureWasmManagedThreads Condition="'$(WasmEnableThreads)' == 'true'">true</FeatureWasmManagedThreads>
+    <FeatureWasmPerfTracing Condition="('$(TargetsBrowser)' == 'true' or '$(TargetsWasi)' == 'true') and ('$(WasmEnableThreads)' == 'true')">true</FeatureWasmPerfTracing>


Is FeatureWasmPerfTracing actually used for anything? We don't really have any tracing support...

SingleAccretion · 2025-07-15T19:24:54Z

src/coreclr/nativeaot/System.Private.CoreLib/src/System.Private.CoreLib.csproj

+    <Compile Include="System\Threading\PortableThreadPool.NativeAot.Browser.cs" Condition="'$(TargetsBrowser)' == 'true' and '$(WasmEnableThreads)' == 'true'" />
+    <Compile Include="System\Threading\ThreadPool.NativeAot.Browser.cs" Condition="'$(TargetsBrowser)' == 'true' and '$(WasmEnableThreads)' == 'true'" />


Why are the portable thread pool files insufficient and we need these as well?

SingleAccretion · 2025-07-15T19:25:53Z

src/native/external/zlib-ng.cmake

+    if (CLR_CMAKE_TARGET_BROWSER)  
      add_compile_options(-pthread)
      add_linker_flag(-pthread)
+    endif()


Per the above comment of not fixing WASI before it can be fixed properly, I think we can remove this condition as well.

cmake seems broken with this for wasi

[vcvarsall.bat] Environment initialized for: 'arm64' Commencing build of native components Not searching for unused variables given on the command line. -- The C compiler identification is Clang 19.1.5 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: c:/github/wasi-sdk25/bin/clang.exe - skipped -- Detecting C compile features -- Detecting C compile features - done -- Performing Test COMPILER_SUPPORTS_W_RESERVED_IDENTIFIER -- Performing Test COMPILER_SUPPORTS_W_RESERVED_IDENTIFIER - Success -- Looking for include files sys/auxv.h, asm/hwcap.h -- Looking for include files sys/auxv.h, asm/hwcap.h - not found -- Looking for sysctlbyname -- Looking for sysctlbyname - not found -- Looking for arc4random_buf -- Looking for arc4random_buf - found -- Looking for O_CLOEXEC -- Looking for O_CLOEXEC - found -- Looking for include file windows.h -- Looking for include file windows.h - not found -- Looking for include files windows.h, bcrypt.h -- Looking for include files windows.h, bcrypt.h - not found -- Looking for clock_gettime_nsec_np -- Looking for clock_gettime_nsec_np - not found -- Using CMake version 4.0.1 -- ZLIB_HEADER_VERSION: 1.3.1 -- ZLIBNG_HEADER_VERSION: 2.2.1 -- Arch detected: 'wasm32-wasip2' -- Basearch of 'wasm32-wasip2' has been detected as: 'wasm32' -- Using CMake toolchain: C:/github/wasi-sdk25/share/cmake/wasi-sdk-p2.cmake -- Performing Test FNO_LTO_AVAILABLE -- Performing Test FNO_LTO_AVAILABLE - Failed -- Looking for arm_acle.h -- Looking for arm_acle.h - not found -- Looking for sys/auxv.h -- Looking for sys/auxv.h - not found -- Looking for sys/sdt.h -- Looking for sys/sdt.h - not found -- Looking for unistd.h -- Looking for unistd.h - not found -- Looking for linux/auxvec.h -- Looking for linux/auxvec.h - not found -- Looking for sys/types.h -- Looking for sys/types.h - not found -- Looking for stdint.h -- Looking for stdint.h - not found -- Looking for stddef.h -- Looking for stddef.h - not found -- Check size of off64_t -- Check size of off64_t - failed -- Check size of _off64_t -- Check size of _off64_t - failed -- Check size of __off64_t -- Check size of __off64_t - failed -- Looking for fseeko -- Looking for fseeko - not found -- Looking for strerror -- Looking for strerror - not found -- Looking for posix_memalign -- Looking for posix_memalign - not found -- Performing Test HAVE_NO_INTERPOSITION -- Performing Test HAVE_NO_INTERPOSITION - Failed -- Performing Test HAVE_ATTRIBUTE_VISIBILITY_HIDDEN -- Performing Test HAVE_ATTRIBUTE_VISIBILITY_HIDDEN - Failed -- Performing Test HAVE_ATTRIBUTE_VISIBILITY_INTERNAL -- Performing Test HAVE_ATTRIBUTE_VISIBILITY_INTERNAL - Failed -- Performing Test HAVE_ATTRIBUTE_ALIGNED -- Performing Test HAVE_ATTRIBUTE_ALIGNED - Failed -- Performing Test HAVE_BUILTIN_ASSUME_ALIGNED -- Performing Test HAVE_BUILTIN_ASSUME_ALIGNED - Failed -- Performing Test HAVE_BUILTIN_CTZ -- Performing Test HAVE_BUILTIN_CTZ - Failed -- Performing Test HAVE_BUILTIN_CTZLL -- Performing Test HAVE_BUILTIN_CTZLL - Failed -- Performing Test HAVE_PTRDIFF_T -- Performing Test HAVE_PTRDIFF_T - Failed -- Check size of void * -- Check size of void * - failed -- sizeof(void *) is bytes CMake Error at C:/github/runtimelab/src/native/external/zlib-ng/CMakeLists.txt:556 (message): sizeof(void *) is neither 32 nor 64 bit

SingleAccretion · 2025-07-15T19:34:30Z

eng/pipelines/common/platform-matrix.yml

@@ -489,6 +489,7 @@ jobs:
      helixQueuesTemplate: ${{ parameters.helixQueuesTemplate }}
      variables: ${{ parameters.variables }}
      osGroup: wasi
+      osSubGroup: ${{ parameters.osSubGroup }}


Needed (also below)?

yowl · 2025-07-15T23:11:30Z

Larger overall point on testing.

We currently don't have even the basic support for e. g. correct memory orderings, so stuff will fail due to race conditions randomly in CI. Should we disable all tests (and only do build) for now, before we know the configuration is at least not 100% known to be flaky? Alternatively, only enable like one test (e. g. HelloWasm)?

Ok, that's fine with me. Will remove in next commit.

Co-authored-by: SingleAccretion <62474226+SingleAccretion@users.noreply.github.com>

…l/runtimelab into FEATURE_WASM_MANAGED_THREADS

jkotas added the area-NativeAOT-LLVM LLVM generation for Native AOT compilation (including Web Assembly) label Apr 27, 2025

jkotas reviewed May 26, 2025

View reviewed changes

yowl closed this Jul 3, 2025

yowl reopened this Jul 3, 2025

Add packages and CI for multithreaded debug config.

494d4c5

yowl force-pushed the FEATURE_WASM_MANAGED_THREADS branch from cba3e94 to 494d4c5 Compare July 5, 2025 12:42

yowl closed this Jul 5, 2025

yowl reopened this Jul 5, 2025

yowl closed this Jul 5, 2025

yowl reopened this Jul 5, 2025

yowl marked this pull request as ready for review July 5, 2025 18:39

SingleAccretion reviewed Jul 5, 2025

View reviewed changes

yowl added 6 commits July 7, 2025 18:20

use nameSuffix in preference to new platforms.

46699a7

Dont emulate for Multithreaded

7fc1d34

balance brackets

e8e2de4

pass WasmEnableThreads to build the packages to get the right name

301c775

set DCLR_CMAKE_TARGET_OS_SUBGROUP in runtime.proj

456f600

Remove CLR_CMAKE_TARGET_OS_SUBGROUP from yml

4099b10

yowl added 5 commits July 11, 2025 16:42

disable PackagingTests for multithread

c3ed443

remove cmakeargs arg from yml and docs.

3261466

add wasmEnableThreadsArg

900df40

Remove PalGetMaximumStackBounds_MultiThreadedWasm and add pthread_att…

8ccf2ec

…r_getstack

revert

d494b8e

SingleAccretion reviewed Jul 15, 2025

View reviewed changes

Update eng/pipelines/runtimelab/runtimelab-post-build-steps.yml

06f4e2d

Co-authored-by: SingleAccretion <62474226+SingleAccretion@users.noreply.github.com>

yowl added 3 commits July 15, 2025 19:09

address feedback for running tests.

2672e75

Merge branch 'FEATURE_WASM_MANAGED_THREADS' of https://github.com/yow…

3b7cfbc

…l/runtimelab into FEATURE_WASM_MANAGED_THREADS

remove D_WASI_EMULATED_PTHREAD

ddf7606

		#elif defined(HOST_WASM) && defined(FEATURE_WASM_MANAGED_THREADS)
		PalGetMaximumStackBounds_MultiThreadedWasm(&pStackLowOut, &pStackHighOut);

		add_compile_options(-D_WASI_EMULATED_PTHREAD)
		add_linker_flag(-Wl,-lwasi-emulated-pthread)

		<Compile Include="System\Threading\PortableThreadPool.NativeAot.Browser.cs" Condition="'$(TargetsBrowser)' == 'true' and '$(WasmEnableThreads)' == 'true'" />
		<Compile Include="System\Threading\ThreadPool.NativeAot.Browser.cs" Condition="'$(TargetsBrowser)' == 'true' and '$(WasmEnableThreads)' == 'true'" />

[NativeAOT-LLVM] Wasm managed threads - build new nupkg #3060

Are you sure you want to change the base?

[NativeAOT-LLVM] Wasm managed threads - build new nupkg #3060

Uh oh!

Conversation

yowl commented Mar 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yowl commented Jul 5, 2025

Uh oh!

SingleAccretion left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yowl commented Jul 11, 2025

Uh oh!

SingleAccretion commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SingleAccretion left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yowl commented Jul 15, 2025

Uh oh!

Uh oh!

yowl commented Mar 30, 2025 •

edited

Loading

SingleAccretion commented Jul 11, 2025 •

edited

Loading