Native memory consumption of ASP.net core application

[Danielku15]

Hi everyone. 👋

I lately did some profiling of our application we migrated to ASP.net core and I noticed quite a big memory consumption for an application like that. First I thought it is our fault because of some custom stuff we're doing, so I ran a memory profiler. But surprisingly it is not our managed code which is allocating this memory, but it is rather native memory that is consuming a lot of space. It is for me not yet sure if it is general ASP.net core or general .net related:

Here the usual graph (left is after startup+force GC, right is after opening the landing page + force GC)

160MB native memory feels like a lot 😱.

In order to find out where this is coming from, I was hoping if somebody can share some insights on what native memory ASP.net core applications might use. My Setup:

• ASP.net core 2.2 (sorry guys, we're working hard migrating our codebase to .net 5 but we have tons of dependencies to sort out) 🙇
• ASP.net MVC with Razor Views
• Running on .net 4.8
• Debug build using Kestrel

So far I have following candidates where I could think our application could use more memory:

• It's the loading of our localization / translation DLLs (our fault) Update: gave it a test to avoid loading all translations, memory consumption is the same. 😞
• Some Caches of dynamically compiled razor pages (we need this)
• Some settings in Kestrel which might reserve some buffers

A plain "Hello World" application crawls around at ~50MB in total which would be fine.

As it is quite hard to find out what native memory is allocated and from where, I thought of reaching out to the community to get maybe some input. I will try to dig deeper in parallel with UMDH and some memory dump analysis tools we have but I fear it will be hard to dig out these details.

Greetings
Daniel

[davidfowl]

@Maoni0 @jkotas any advice for a better understanding of memory usage here (this application is running on .NET Framework) ?

Server GC is likely one reason for the memory usage and dynamic compilation. Using roslyn in process at runtime is likely a reason for high memory usage.

[jkotas]

Profile it using a tool that one is comfortable with. UMDH should be a fine start.

[Danielku15]

Thanks, I will dig a bit deeper with UMDH and my home-brew Memory Dump analyzer (it scans the memory of full-mem dumps for VTables and counts objects by name using it). Am I right with the assumption that there are no PDBs available for the .net 4.8 which would allow me to properly use UMDH? Allocation stacks are only properly resolved if you have them. I might be able to see "something" but I fear it might be hard to answer what exactly is allocated.

[jkotas]

Public Microsoft symbol servers should have PDBs for net4.8.

[Danielku15]

Update:

The initial growth from 0 to ~80MB (with 54MB native memory) mainly seem to be caused by some assembly loading. I see a lot of allocates in relation to PE file (and PDB?) loading and some internal allocations. We have an overall of 163 assemblies in our project (22 System., 106 Microsoft., + our nuget and own dlls) which sum up to 32MB DLLs + 7MB PDBs on Disk.

The second grow seems again loading a lot of assemblies, types, etc. Also some JIT compilation. Quite hard to judge for me if this is caused by ASP.net core specifics or .net in general. 😟

I have uploaded (parts) of the UMDH diffs to this gist: https://gist.github.com/Danielku15/c2139b453a66da62d529bfb3a745071c

Somebody has an idea here?

We have ~60 controllers in our project, and ~200 cshtml files (layouts, partials, views, all included) if this gives some indication.

[jkotas]

Do the allocations reported in the UMDH log add up to anywhere 117MB?

https://docs.microsoft.com/en-us/sysinternals/downloads/vmmap tool gives better top-level breakup of the memory. It reports unmanaged heaps, managed heaps (GC heap and code/loader heap) and the rest. You can give it a try.

[Danielku15]

Do the allocations reported in the UMDH log add up to anywhere 117MB?

No they don't. UMDH only reports a growth of 24MBs (which fits roughly the numbers on my testrun below).

https://docs.microsoft.com/en-us/sysinternals/downloads/vmmap tool gives better top-level breakup of the memory

I did another trace session and collected again the usage with dotMemory and VVMap

After Startup

After Page open

dotMemory claims that the unmanaged memory goes from ~53MB to ~118MB (+65MB).
In the Private WS of VMMap I can see that the process memory goes from 39MB to 122MB (+83MB)

The biggest growths are in:

1. The normal heap goes from 11MB to 48MB (+37MB)
   
2. The Managed Heap goes from 21MB to 57MB. (+36MB)
   

dotMemory claims that there are only 26MB of objects around and the heaps have no significant fragmentation or unused space:

It's tricky to get all the things together at the right time as they are easily lost during testing. I will try to again do a full testround with all tools active (dotMemory, UMDH, VMMap) and saving all details.

Does anybody maybe already have a clue where this memory could come from?

[Danielku15]

So, this time I can share some more insights and conclusions with a bit more structure. I followed my usual testing procedure of:

1. Launch the app (via dotMemory)
2. Force GC (via dotMemory)
3. Collect all details (VMMap, UMDH, Snapshot in dotMemory)
4. Open the landing page in the browser
5. Force GC
6. Collect all details (VMMap, UMDH, Snapshot in dotMemory)

The general memory increase can be seen in this table:

dotMemory does not help so much this time on the native part because UMDH requires quite a bunch of additional memory. VMMap shows the typical increase of memory I am trying to catch. An increase of 34MB on the managed side and 49MB on the native side is more than we would expect from our application.

Looking the managed objects I can see that the main memory comes from two parts.

• Many strings related to razor view compilations.
• A lot of type and method (reflection) data being loaded for the ASP.net core infrastructure.
  I am not sure how much I can optimize here but I will try to play with some tricks to maybe not load so many DLLs and symbols (e.g. by adapting the DefaultApplicationModelProvider and doing some more pre-compilation tricks).

For the native heap part I also tried to dig a bit deeper to find out where this memory comes from. In general I can find that some heap blocks are new and filled with a lot of data:

It's another 35MB native memory on top of the managed memory. UMDH tells me also that there are additional 38MB allocated.

23MB are from this stack trace:

+ 16d5768 ( 16d5768 -     0)    113 allocs	BackTrace1CD857FA
+     113 (   113 -     0)	BackTrace1CD857FA	allocations

	ntdll!RtlpAllocateHeapInternal+96D77
	KERNELBASE!LocalAlloc+6D
	mscorlib.ni!System.Runtime.InteropServices.Marshal.AllocHGlobal(IntPtr)$##600600F+69
	System.Reflection.Metadata.ni!???+0 : 7FFA69ED20F9
	System.Reflection.Metadata.ni!???+0 : 7FFA69ED20D4
	<no module>!???+0 : 7FFAA0736B17

The significant allocates mainly come from the profiling interface and tools which were causing additional memory load in the process.

If I am interpreting this right the main memory load in the native part comes from similar things like on the managed part: Loading of DLLs, reflection info etc.

At least I have now 2 action points that should allow some more fiddling on getting the load down:

1. Try to set up fully pre-compiled razor pages to avoid dynamic runtime compilation costs.
2. I will try to reduce the allocates caused from the PE file loading and scanning by being more specific where ASP.net core can find its relevant parts.

As a reference here some backtraces where I can see quite a lot of object allocates. They might not make up so much effective bytes (we have only 25MB of managed objects in total), but seeing the UMDH allocates and other numbers I guess they also cause the native allocations indirectly (e.g. via native images).

Thanks so far for all your help, if you have more ideas or hints I am happy to hear them. I will try to share my insights if I really manage to get the consumption down.

[davidfowl]

The routing logic is much more optimized on ASP.NET Core 3.0 but you're on .NET Framework so that's not a solution for you 😄

[Danielku15]

We have a quite complex service oriented architecture which is heavily built on WCF. Unfortunately for some things we've been using in WCF (e.g duplex contracts) there is no successor technology available 😭 Hence we are forced to redevelop/-design a lot of our communication architecture. Investing so much time+money just that it works like before is hard to sell to management.

[Danielku15]

Good news on the native memory consumption. I was able to identify the main cause of it, it is really the Razor compilation.

The Razor SDK is effective only when no precompilation-specific properties are set in the project file. For instance, setting the .csproj file's MvcRazorCompileOnPublish property to true disables the Razor SDK.
https://docs.microsoft.com/en-us/aspnet/core/mvc/views/view-compilation?view=aspnetcore-2.2&tabs=visual-studio

So far I was playing with setting the MvcRazorCompileOn** to true or false and could not see a difference. But when removing the tags like described above, things changed! This is the memory consumption after removing the tags fully:

Adding them again with "true" leads to:

The yellow part is the part that stays the same. The red part is the additional cost of the Razor compilation. I would say the yellow consumption is on a level which is acceptable to an application of our size (based on gut-feeling).

Now I will need to evaluate the risk of not being able to adjust razor files on demand in installations. It helps sometimes on error investigations. Best would be to have something "combined" where it uses the DLL by default and only compiles changes if needed.

[Danielku15]

I was able to improve the memory consumption to a reasonable level. To fully answer my initial question:

The main memory consumption is coming from the Razor file compilation. We were not using any precompiled views befoer which caused Roslyn to load a lot of things and perform a lot of tasks which caused the load. It is unclear why this is detected as Native Memory, but it could be related to some dynamic PE file generation and loading happening there.

To reduce the consumption without losing the possibility to dynamically change Razor templates following things needed to be changed:

1. According to https://docs.microsoft.com/en-us/aspnet/core/mvc/views/view-compilation?view=aspnetcore-2.2&tabs=visual-studio no precompile tags have to be in the project file. not even one with false seem to be critical. so step one was to remove all MvcRazorCompileOn** tags.
2. To not lose the razor file deployment and dynamic changing of files there need to be some additional settings in the file (ensures the deployment of the files and that the view compilation is enabled). Changing of the files still requires an IIS/AppPool/Process restart to make the changes take place, but that's OK for us.

<PropertyGroup>
  <MvcRazorExcludeViewFilesFromPublish>false</MvcRazorExcludeViewFilesFromPublish>
  <MvcRazorExcludeRefAssembliesFromPublish>false</MvcRazorExcludeRefAssembliesFromPublish>
</PropertyGroup>
<ItemGroup>
  <PackageReference Include="Microsoft.AspNetCore.Mvc.Razor.ViewCompilation" Version="2.2.0" />
</ItemGroup>

Hope this helps others having similar issues.

[davidfowl]

It is unclear why this is detected as Native Memory, but it could be related to some dynamic PE file generation and loading happening there.

It's because internally Roslyn uses System.Reflection.Metadata which uses native memory.