Why increasing of "Unprocessing Messages" leads to increasing nats memory usage?

[tierpod]

Hello. I noticed this behaviour sometimes and don't know how to explain it.

We have 1 stream configured with disk storage:

             Replicas: 3
              Storage: File

Options:

            Retention: WorkQueue
     Acknowledgements: true
       Discard Policy: Old
     Duplicate Window: 2m0s
    Allows Msg Delete: true
         Allows Purge: true
       Allows Rollups: false

Limits:

     Maximum Messages: unlimited
  Maximum Per Subject: unlimited
        Maximum Bytes: 10 GiB
          Maximum Age: 2d0h0m0s
 Maximum Message Size: 2.0 MiB
    Maximum Consumers: unlimited

State:

             Messages: 2,581,362
                Bytes: 6.7 GiB
             FirstSeq: 7,993,228,463 @ 2025-04-21T13:22:50 UTC
              LastSeq: 8,171,038,132 @ 2025-04-23T10:19:48 UTC
     Deleted Messages: 175,228,308
     Active Consumers: 10
   Number of Subjects: 10

At some time, consumer fails during deployment for a while (broken deploy, for example) and stops to consume messages. Publisher continues to publish messages. As a result, stream data size grows, "Unprocessed Messages" in nats consumer info grows.

My expectation

It's OK that data size grows in that case, because nats saves unprocessed messages to disk. "Unprocessed Messages" counter increases. Host memory usage should not increase significantly (because messages saved to stream on disk, not in memory). After consumer fixes problems and continues to consume messages from stream, "Unprocessed Messages" should decreasing.

In fact

Increasing value of "Unprocessing Messages" leads to host memory usage increasing, and I don't understand why

[neilalexander]

Can you please post the consumer infos as well?

[tierpod]

Configuration:

                Name: android
           Pull Mode: true
      Filter Subject: Events.android
      Deliver Policy: All
          Ack Policy: Explicit
            Ack Wait: 11m40s
       Replay Policy: Instant
     Max Ack Pending: 56,000
   Max Waiting Pulls: 512

Cluster Information:

                Name: jetstream
              Leader: nats01
             Replica: nats02, current, seen 0.18s ago
             Replica: nats03, current, seen 0.18s ago

State:

   Last Delivered Message: Consumer sequence: 5,822,253,628 Stream sequence: 8,183,813,928 Last delivery: 0.08s ago
     Acknowledgment floor: Consumer sequence: 5,822,248,190 Stream sequence: 8,183,806,285 Last Ack: 0.97s ago
         Outstanding Acks: 3,932 out of maximum 56,000
     Redelivered Messages: 0
     Unprocessed Messages: 2,002,566
            Waiting Pulls: 0 of maximum 512

On the graphics above Unprocessed messages value increased from ~0-1000 to 2,800,000. Now I see that Unprocessed messages decreasing gradually and memory usage decreasing as well.

[MauriceVanVeen]

Feel like the Deleted Messages being over 175 million is more likely to be what's holding the memory.

[tierpod]

Interesting, I didn't think about about it. Is there any logic that makes nats hold deleted message in memory? I thought that this is some kind of counter: consumer reads message from WorkQueue stream, acknowledges it, and after that message is deleted from stream and counter increased.

[MauriceVanVeen]

Deleted Messages are "interior" deletes. Let's say you have a stream with 3 messages, first sequence is 1 and last sequence is 3. You have no deleted messages, so that would be 0. When you remove/ack message at sequence 2, then you'll have 1 interior delete, so Deleted Messages 1.

When your consumer came back and started processing it decreased the unprocessed messages (logically), but it also cleaned up all these interior deletes.

[tierpod]

Thank you for clarification! It's very important to understand.

[tierpod]

Another day, another high memory usage

             Messages: 846,944
                Bytes: 2.1 GiB
             FirstSeq: 8,079,883,098 @ 2025-04-22T11:35:33 UTC
              LastSeq: 8,276,773,207 @ 2025-04-24T10:24:14 UTC
     Deleted Messages: 196,043,166

I see that Deleted messages higher that yesterday (196,043,166 vs 175,228,308). I ran nats str info multiple times and see that Deleted Messages changed, but FirstSeq didn't:

             Messages: 987,371
                Bytes: 2.4 GiB
             FirstSeq: 8,079,883,098 @ 2025-04-22T11:35:33 UTC
              LastSeq: 8,277,928,219 @ 2025-04-24T10:35:04 UTC
     Deleted Messages: 197,057,751

So I assume, it is a problem - some consumer keeps message from the beginning of the stream (that's why FirstSeq doesn't change), and nats cannot delete it from disk (and keeps information about all the Deleted Messages in memory).

[derekcollison]

We keep track of interior deletes by sequence number in a bit mask data structure. So efficient but at those numbers will have an impact on memory use to some degree.

[wallyqs]

How big are the messages in general?

[tierpod]

According to our estimates, most messages should have size less than 2kb. One type of messages (from one service) can have size 350~500kb. It's hard to estimate, is there any way to connect to consumer and listen conversation, without affecting services?

[wallyqs]

thanks, you could connect using a raw nats connection on nats sub '$JS.API.CONSUMER.MSG.NEXT.>' for a brief time to confirm and get an idea, the message size would be written in the protocol control line on the response.

[wallyqs]

I was wrong here it should have been $JS.ACK.> since we are looking for the size of the message, above was the fetch request

[tierpod]

I tried but with no success. I see nothing about message size in both outputs:

[#18] Received on "$JS.API.CONSUMER.MSG.NEXT.Events.android" with reply "_INBOX.J21sYfFwAMufdJCDyuP9W3"
{"expires":5000000000,"batch":1,"idle_heartbeat":2500000000}

[#2075] Received on "$JS.ACK.Events.android.1.340208729.196022044.1745485966373073386.0"
+ACK

[wallyqs]

Could you share how you are consuming the messages (which API calls)? Are you using MaxBytes when fetching?

[tierpod]

We are using dotnet library, FetchAsync call, MaxBytes not set

[wallyqs]

I would try to set a MaxBytes limit because I feel that maybe your client is asking for too many messages (up to 56000 which is your max acks inflight) when it is behind which will make the server allocate many buffers at once since the server tries to reuse the buffers whenever possible and could take a long time for this memory used to decrease once allocated. You can try setting MaxBytes to something like 8MiB or 16MiB (but not less than 2MB since you could have messages that size, and definitely not less than the max_payload that you are setting in the server).

[wallyqs]

re: Maximum Message Size: 2.0 MiB, how often do you have messages over 1MB?

[tierpod]

We reached this limit (1 MB) once or twice in special cases, so we increased it. In general, messages should not reach the 1 MB limit.

[tierpod]

I gathered more information, if someone interested