[wasm] High memory fragmentation on LOS allocations

[andrew-kulikov]

Description

I am porting a huge .NET application (a game written on custom engine) to browser-wasm runtime and noticed that heap fragmentation is pretty high. The initial idea was that this may be caused by major heap blocks fragmentation. I tried to play with MONO_GC_PARAMS increasing/decreasing nursery-size, evacuation-threshold and soft-heap-limit, and triggering gc frequently. But there was no noticeable difference in FragmentedBytes metric.

Then I used emscripted memory profiler and found that there is a lot of allocation with stack traces like this:

at dotnet.native.wasm.dlmalloc (http://localhost:8081/_framework/dotnet.native.wasm:wasm-function[12628]:0x29d1ca)
at dotnet.native.wasm.internal_memalign (http://localhost:8081/_framework/dotnet.native.wasm:wasm-function[12634]:0x29e12b)
at dotnet.native.wasm.dlposix_memalign (http://localhost:8081/_framework/dotnet.native.wasm:wasm-function[12635]:0x29e2d8)
at dotnet.native.wasm.mono_valloc_aligned (http://localhost:8081/_framework/dotnet.native.wasm:wasm-function[931]:0x4af61)
at dotnet.native.wasm.sgen_alloc_os_memory_aligned (http://localhost:8081/_framework/dotnet.native.wasm:wasm-function[1508]:0x6b380)
at dotnet.native.wasm.sgen_los_alloc_large_inner (http://localhost:8081/_framework/dotnet.native.wasm:wasm-function[1424]:0x6094c)
at dotnet.native.wasm.sgen_alloc_obj_nolock (http://localhost:8081/_framework/dotnet.native.wasm:wasm-function[1280]:0x57658)

I prepared a sample repo with reproduction. The sample is named "Los fragmentation", and it allocates a lot of byte arrays of configured size. After playing with parameters I found that if array size is greater than 7984 (SGEN_MAX_SMALL_OBJ_SIZE - mono object size), fragmentation grows significantly. Sample code:

[JSExport]
public static async Task LosFragmentation(int count, int batchSize, int size)
{
    var losHolder = new List<byte[]>();
    for (int i = 0; i < count; i++)
    {
        PrintMemory($"LF #{i} - start");

        for (int j = 0; j < batchSize; j++)
        {
            losHolder.Add(new byte[size]);
        }
        await Task.Delay(16);
        GC.Collect();
        PrintMemory($"LF #{i} - end");
    }
}

Here is output with different parameters:

1. count=10, batchSize=1000, size=8192
LF #9 - end | Total Memory: 82.37 MB | HeapSizeBytes: 236.64 MB | FragmentedBytes: 158.29 MB | TotalCommittedBytes: 78.35 MB | G0: 0 | G1: 10 | G2: 10 | 

1. count=10, batchSize=1000, size=7984
LF #9 - end | Total Memory: 80.39 MB | HeapSizeBytes: 80.77 MB | FragmentedBytes:  | TotalCommittedBytes: 84.00 MB | G0: 10 | G1: 10 | G2: 10 | 

Configuration

.NET 10.0.100-preview.5.25277.114
Chrome 138.0.7204.158

Data

7984:

8192:

Workaround

Please explain if it is expected behavior, are there any known workarounds (including building custom runtime) and is there any planned work on this issue.

[dotnet-policy-service]

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @BrzVlad
See info in area-owners.md if you want to be subscribed.

[BrzVlad]

I think #108512 is related. I believe the issue is caused by los sections requiring massive alignment. A trivial fix could be to disable los sections on wasm and always allocate large objects directly with posix_memalign. @pavelsavara What do you think ?

[pavelsavara]

if array size is greater ... fragmentation grows significantly.

Yes

I think #108512 is related.

Yes, that change improved fragmentation from even worse state.

I believe the issue is caused by los sections requiring massive alignment.

Yes

A trivial fix could be to disable los sections on wasm and always allocate large objects directly with posix_memalign.

I'm not confident about internal details of sgen GC, but I think it uses that alignment to create some bit "map" of pages that the GC uses. And that low bits of pointer carry some meaning. Perhaps I'm confused.

Maybe we could fiddle with SGEN_MAX_SMALL_OBJ_SIZE

Maybe developers could use Marshal.AllocHGlobal instead of byte[] for large data.

[BrzVlad]

We have 2 allocators for large objects in the GC. Either in los sections or directly from the OS (for even larger objects). We could allocate directly from the OS (posix_memalign) all the time, assuming this call is not extremely slow for whatever reason. We don't really have alignment requirements for los objects. I think changing SGEN_MAX_SMALL_OBJ_SIZE is way more invasive and riskier and still doesn't address the reason behind fragmentation of los objects.

[pavelsavara]

posix_memalign assuming this call is not extremely slow

I think it's not too slow.
Asking for aligned memory will try to find free page in dlmalloc and if there is none, it will sbrk new address space.

https://github.com/emscripten-core/emscripten/blob/afb59ffdd968f445e0b4b11ffb69548ed67543fe/system/lib/dlmalloc.c#L4984

• wasm doesn't have mmap/unmap (you can't return memory)
• only 32bit address space (that's easily fragmented)
• address space also only knows how to grow (no negative sbrk).

We don't really have alignment requirements for los objects. I think changing SGEN_MAX_SMALL_OBJ_SIZE is way more invasive and riskier and still doesn't address the reason behind fragmentation of los objects.

OK

We have 2 allocators for large objects in the GC

Where/how to use the other one ?