Understanding jetstream warnings

[astelmashenko]

Hey, Community,
We are using Nats/Jetstream 2.9.19 (on alpine), three nodes like below:

 NAME                                  READY   STATUS    RESTARTS   AGE
jetstream-nats-0                      4/4     Running   0                   89m
jetstream-nats-1                      4/4     Running   0                   3d7h
jetstream-nats-2                      4/4     Running   0                  2d18h
jetstream-nats-box-6bf4bcbc4f-rssf8   2/2     Running   0          3d6h

We noticed an application failed to publish a message with timeout error and the same time a lot of messages on jetstream:

[61] 2025/02/08 15:25:27.467719 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:25:41.480542 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:25:47.005060 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:25:52.530745 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:25:58.054230 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:26:03.567590 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:26:40.490602 [INF] JetStream cluster new consumer leader for '$G > view_configs_change > zU6ZviET'
[61] 2025/02/08 15:27:00.083845 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:27:05.972497 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:27:11.495172 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:27:17.009565 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:27:22.535667 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:27:28.047313 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:28:02.647994 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag

We are using AWS EBS disks as storage. There were no high load or anything extraordinary.

Only restart helped to resolve that. Any thoughts on what could be the problem?
What does "has high message lag" mean?
If there is something I need to add about our system, please let me know.

[rajeshkarka]

I am also facing same issue and publisher is getting slow, is this because of number of replicas in cluster nodes?
We are running 5 node cluster and created stream with 3 replica's.

We are seeing "high message lag" warning and publisher is becoming slower to push the traffic. How can I avoid this problem?

[derekcollison]

This means you are sending faster then the system can process and store messages into the stream. This can happen if you use a core publish into a stream or if you use async Jetstream publishes with many publishers.