GC never collects anything, help me understand

[alex-jitbit]

I'm trying to understand high memory usage of an ASP.NET app (net8), looking at these dotnet-counters results:

[System.Runtime]
    % Time in GC since last GC (%)                                         0
    Allocation Rate (B / 1 sec)                                    9,004,088
    CPU Usage (%)                                                         16.418
    Exception Count (Count / 1 sec)                                       30
    GC Committed Bytes (MB)                                            3,961.729
    GC Fragmentation (%)                                                  47.313
    GC Heap Size (MB)                                                  2,015.54
    Gen 0 GC Budget (MB)                                               5,368
    Gen 0 GC Count (Count / 1 sec)                                         0
    Gen 0 Size (B)                                                    6.7017e+08
    Gen 1 GC Count (Count / 1 sec)                                         0
    Gen 1 Size (B)                                                    3.2299e+08
    Gen 2 GC Count (Count / 1 sec)                                         0
    Gen 2 Size (B)                                                    2.8457e+08
    IL Bytes Jitted (B)                                            9,169,971
    LOH Size (B)                                                      2.3909e+08
    Monitor Lock Contention Count (Count / 1 sec)                         15
    Number of Active Timers                                            1,556
    Number of Assemblies Loaded                                          229
    Number of Methods Jitted                                          88,840
    POH (Pinned Object Heap) Size (B)                             61,413,664
    ThreadPool Completed Work Item Count (Count / 1 sec)               2,481
    ThreadPool Queue Length                                                1
    ThreadPool Thread Count                                                8
    Time paused by GC (ms / 1 sec)                                         0
    Time spent in JIT (ms / 1 sec)                                        22.859
    Working Set (MB)                                                   4,571.111

Running on a 16gb RAM Linux machine

As you can see the app consumes 4.6GB of RAM.

The LOH size is only 239 M, Gen2 is also small, so it's not the "Large objects" that causes high memory usage.

I ran dotnet-trace with gc-collect profile for 30 seconds and found the Garbage collection never happens, ever.

If I keep the app running the mem usage will grow to 7, 8,10,12GB...

I guess the GC is waiting for the "Gen 0 Budget" to fill up and then trigger the collection? The "budget" is set to 5 gigabytes, however gen0 is using only 670MB, so it will take a while to get there.

Running the same app on a smaller server with twice less memory consumes only 4GB or RAM and stays there. So I'm assuming the GC heuristics is simply "oh we have plenty or memory, let's not do anything". Is there a way I can decrease the "budget"? Or may be it would make sense to switch to "workstation" GC mode?

P.S. Can't really take a process dump cause this is a live production app. However I guess I don't need to, memory grows b/c GC never collects.

UPDATE

After switching to "Workstation" GC mode, the memory usage grows to 1GB and sits there. So I guess that helped. However, I'm not sure that "Workstation" GC mode is the best solution for a high-load web app.

I'm not sure what to do now. Leave it at workstation and experience higher CPU load caused by more frequent GC interruptions, or switch back to "Server" and pray that my app won't be shut down by Linux OOM killer? (already happened a couple of times)

[alex-jitbit]

Closing as I found another workaorund.

[gfoidl]

Please report about that workaround, so it may a) help others b) maybe make the report actionable. Thanks in advance.

[alex-jitbit]

@gfoidl turned out it was an issue with a 3rd party lib that had a memory leak (ever-growing collection), updating that dependency fixed the issue.