How does MaxBatch work with Streaming pull consumer & info on default consumerConfig values

[YashasAnand]

I have the below consumer config, I was operating under assumption that maxBatch is a property used by fetch specific C# consumers that make a pull request to server for messages.

but a per https://nats-io.github.io/nats.net/documentation/jetstream/consume.html its clear that the ConsumeAsync is the most performant of the 3 consume options, so by setting MaxBatch of lets say 1000, does that mean it tries to pull or consume 1000 messages from stream?

also what is the default value if not set?

Can anyone help me out regarding what are the default values set for the ConsumerConfig classes as im unable to find any document which gives these details.

In my current enviroment i get a high request rate & i observed lot of messages being stuck in unprocessed state in stream as MaxAckPending was not initially set on my stream & it defaulted to 1000, it would be nice to know what are the defaults that the system configures if not specified in consumerConfig.

Lets assume i configure maxBatch as 100 & i also use maxWaiting to 10 does that mean the consumer makes 10 pull requests at once in batches of 100?

PS: ive been able to reproduce this issue #452 when i gave maxBatch as 10.

When i set it to 1000 it works , does this mean each pull request will end up receiveing upto 1000 messages, also will this cause issues related to cpu or memory usage than leaving it default?

what happens if i leave MaxBatch as not defined, and as per above issue it states NatsJSConsumeOpts.MaxMsg is being set to 1000 , does this mean for each message a pull request is made which is not being optimized as we have made maxBatch to 1000 => 1 pull request can get upto 1000 messages?

Also can anyone advice for achiveing 100% message delivery without many messages getting stuck in pending or in unprocessed state, what are the "must set" parameters in consumerConfig & what kind of values is recommended for these fields.

var ConsumerConfig = new NATS.Client.JetStream.Models.ConsumerConfig
               {
                   Name = consumerName,
                   DurableName = consumerName,
                   AckPolicy = ConsumerConfigAckPolicy.Explicit,
                   DeliverPolicy = ConsumerConfigDeliverPolicy.New,
                   FilterSubject = partitionSubjectFilter,
                   AckWait = TimeSpan.FromMinutes(_consumer.AckWaitTimeSpanInMinutes),
                   Metadata = new Dictionary<string, string>
                   {
                       {"ConsumerID", Guid.NewGuid().ToString()},  // Unique ID for consumer
                       {"LastMsgDeliveredTime", DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss",CultureInfo.InvariantCulture)} // Consumer Last delivered Msg Time
                   },
                   MaxDeliver = 3,
                   MaxAckPending = 50000,
                   MaxBatch = ?

                   

               };
               INatsJSConsumer jsConsumer = await _natsJSContext!.CreateOrUpdateConsumerAsync(_consumer.ConsumerAssociatedStream, ConsumerConfig).ConfigureAwait(false);
               _logger($"Started Consumer {_consumer.ConsumerTopic} with FilterSubject {partitionSubjectFilter}",Levels.Info);
               await foreach (var jsMsg in jsConsumer.ConsumeAsync(serializer:natsMesgSerializer,cancellationToken:_cts.Token))
               {
               //Process consumer message
               }

[mtmk]

By default consume() sets MaxMsgs (batch) to 1000. This value determines how many messages are expected from the server in every batch. This is called a pull request. With this flow control mechanism, client issues pull requests and server fulfils the request either by sending the amount of messages asked for or until the request times out, which is 30 seconds by default. One trick client does is to overlap the pull requests using the threshold settings (these default to half the batch size). When a threshold reached, client issues another pull request to keep the flow going uninterrupted.

In your case probably setting a lower batch value would help. it depends on how you're processing the messages, how long the processing takes, failure rate etc. Best way i found is to experiment in a test environment to find what best suits you. You can find explanation of the consumer settings here: https://docs.nats.io/nats-concepts/jetstream/consumers#pull-specific

edit: defaults are..

documented here:

nats.net/src/NATS.Client.JetStream/INatsJSConsumer.cs

Line 18 in 1827eb5

/// <param name="opts">Consume options. (default: <c>MaxMsgs</c> 1,000)</param>

set here:

nats.net/src/NATS.Client.JetStream/NatsJSOpts.cs

Line 51 in 7af473c

public NatsJSConsumeOpts DefaultConsumeOpts { get; init; } = new() { MaxMsgs = 1_000 };

[YashasAnand]

Thank you so much for your help and advice @mtmk