Is it normal for a .NET application to use more memory on ARM compared to x64?

[asliazas]

Hi,

I've noticed that when running the same .NET application on an ARM platform, memory usage appears to be significantly higher than on an x64 platform. Here's an example showing increased memory usage after migrating to an ARM-based AWS Graviton instance:

To investigate further, I created a minimal .NET application and ran it inside a Docker container on both AMD64 and ARM64 platforms. Here’s the memory usage comparison:
AMD64:

ARM64

You can find the test app here:
IncreasedMemoryTestApp.zip

Steps to Reproduce:

1. Enable Docker's BuildKit and QEMU emulation:

docker run --privileged --rm tonistiigi/binfmt --install all

2. Create a multi-architecture builder:

docker buildx create --name mybuilder --driver docker-container --use
docker buildx inspect --bootstrap

3. Build the ARM image:

docker buildx build --platform linux/arm64 -t increased-memory-app:arm64 -f IncreasedMemoryTestApp\Dockerfile .

4. Run the ARM image:

docker run --platform linux/arm64 -p 8080:8080 increased-memory-app:arm64

Is it expected for .NET applications to use more memory on ARM64 than on x64?
Are there known differences in memory management or GC behavior on ARM that could explain this?

Also, any guidance on how to properly determine soft and hard memory limits in such cases to prevent out-of-memory exceptions would be greatly appreciated.

Thanks!

[jkotas]

Does the ARM-based AWS Graviton instance have the same specs as the x64 instance? Same number of cores and same amount of available memory? GC tends to take advantage of available resources when it is beneficial. If you run on a machine with more cores, GC may decide to use more memory to keep all cores occupied.

cc @mangod9 @richlander

[asliazas]

Previously we used m5a.large

Now we switched to m6g.large

Specs looks basically the same

[mangod9]

There might also be a difference in L3 cache sizes between the different hardware configurations, which affect some internal GC heuristics to help optimize allocation costs. It might be worth setting DOTNET_GCgen0size=1E00000 env. var on both configs to check if that makes things comparable with x64.

Some considerations:

1. Please note that it's always a tradeoff between memory utilization and throughput.
2. Have you tried this on .NET 9? On 9 GC is optimized for better memory utilization than earlier versions (assuming you are running on 2+ cores and using SVR gc).

[asliazas]

Setting DOTNET_GCgen0size=1E00000 makes no difference, also we already using .NET 9

[grietine]

Any further ideas? More than >2x memory usage is expected and is going to be the new norm?

[mangod9]

No it should not be the norm, this is still on our radar to investigate.

[EgorBo]

@grietine so I've just checked the repro on m6g.large vs m5a.large - both consumed ~70mb (and the healthcheck endpointed worked). Although, I didn't use the proposed docker workflow, I just launched the processes via this script: https://gist.github.com/EgorBo/c5b6fb9f0859d5a36727137ab7d6255d

[mangod9]

Adding @mrsharm as well, since he has been looking at a similar issue.

[KeterSCP]

Just want to share my observations:

When running containers via QEMU emulation, the emulated architecture (non-host one) consumes a lot more RAM than native one.
For example, here is the output of docker stats for 2 containers of the same multi-arch image (taken from the author repro steps above) running on amd64 host with limits --cpus=2 -m=200m:

And when running it natively on arm64 host (RPI 4B):

So it may not be the best idea to test memory issues on containers with an emulated architecture

[grietine]

@grietine so I've just checked the repro on m6g.large vs m5a.large - both consumed ~70mb (and the healthcheck endpointed worked). Although, I didn't use the proposed docker workflow, I just launched the processes via this script: https://gist.github.com/EgorBo/c5b6fb9f0859d5a36727137ab7d6255d

You have run natively AWS -> .NET App. What about docker on top AWS -> Docker -> .NET App?

@KeterSCP can this be somehow related to the problem, any thoughts?

• Building on Apple M-series processor via Colima?
• Any of the used NuGet's or Native Libraries (.so) are being emulated form x86_64 platform? Which makes the process run, but in emulated way as this QEMU kicks-in? If yes, I thought that it should fail with "DllNotFoundException or similar" error.

[KeterSCP]

@grietine without proper repro its hard to tell anything. I would suggest trying to minimally recreate the issue on containers with fixed memory/cpu limits that have same architecture as the host.