Latency is increasing with load

[debbyglance]

We are running ./nats latency --server-b localhost:4222 --rate 500000 while our system is unloaded, somewhat loaded, and heavily loaded. HDR percentiles are below. Latency increases with load, so there seems to be some sort of bottleneck.

There are 2 nodes in the nats cluster, running on AWS c6i.large instance types

CPU peaks at 25% on the nodes so there does not seem to be a CPU bottleneck.

1. Should we expect latency to remain constant with load, if the cluster is sized correctly?
2. IIf yes, how can we tell where the bottleneck might be?

14:18:27 ==============================
14:18:33 HDR Percentiles:
14:18:33 10: 567µs
14:18:33 50: 3.803ms
14:18:33 75: 7.176ms
14:18:33 90: 11.094ms
14:18:33 99: 21.483ms
14:18:33 99.9: 27.683ms
14:18:33 99.99: 31.52ms
14:18:33 99.999: 31.856ms
14:18:33 99.9999: 31.872ms
14:18:33 99.99999: 31.873ms
14:18:33 100: 31.873ms

15:12:46 ==============================
15:12:51 HDR Percentiles:
15:12:51 10: 1.29ms
15:12:51 50: 5.353ms
15:12:51 75: 9.363ms
15:12:51 90: 13.601ms
15:12:51 99: 24.448ms
15:12:51 99.9: 34.195ms
15:12:51 99.99: 36.74ms
15:12:51 99.999: 36.849ms
15:12:51 99.9999: 36.981ms
15:12:51 99.99999: 36.999ms
15:12:51 100: 36.999ms

16:08:21 ==============================
16:08:27 HDR Percentiles:
16:08:27 10: 1.214ms
16:08:27 50: 5.296ms
16:08:27 75: 10.806ms
16:08:27 90: 16.839ms
16:08:27 99: 30.484ms
16:08:27 99.9: 37.646ms
16:08:27 99.99: 45.332ms
16:08:27 99.999: 45.739ms
16:08:27 99.9999: 45.756ms
16:08:27 99.99999: 45.758ms
16:08:27 100: 45.758ms

[derekcollison]

How is the "load" applied to the system?

[debbyglance]

We are running our service, which utilizes the NATS cluster. Presumably we would see the same degradation had we introduced a similar load via NATS bench. Is this normal, or an indication that we need to scale the cluster out? Maybe the better question is,

1. what is a good methodology for determining whether our cluster is correctly sized?
2. what are options for autoscaling a NATS cluster (we use AWS ECS not Kubernetes)

[derekcollison]

Those questions are nuanced, best to schedule some time with Synadia experts to discuss.