Inconsistent Connection Reporting and Routes in NATS JetStream Cluster Across Pods (Bitnami Helm Chart 8.5.2)

[grok-rs]

I am running a NATS JetStream cluster (version 2.10.24) deployed with the Bitnami Helm Chart (version 8.5.2) on Kubernetes. The setup spans three availability zones (AZs) with three replicas: nats-0, nats-1, and nats-2

Issue 1: Inconsistent Connection Reporting

When I initially start the cluster and run the following command:

nats --server some-dns-name:4222 --user nats --password super_password server report connections

I correctly see three servers (nats-0, nats-1, nats-2) in the cluster, with one of them (e.g., nats-1) elected as the leader. However, when I kill the leader pod (nats-1), a new leader is elected (e.g., nats-0). After this, if I proxy either nats-0 or nats-2 using:

kubectl port-forward nats-0 4222:4222 -n nats

nats --server some-dns-name:4222 --user nats --password super_password server report connections

... streams...
╭────────────────────────────────╮
│     Connections per server     │
├────────┬─────────┬─────────────┤
│ Server │ Cluster │ Connections │
├────────┼─────────┼─────────────┤
│ nats-0 │ nats    │           1 │
│ nats-2 │ nats    │         240 │
╰────────┴─────────┴─────────────╯

and re-run the same command, I see only two servers in the cluster, even though the nats-1 pod is running. However, if I proxy nats-1 and re-run the command, I see all three servers again.

Is this expected behavior for NATS JetStream clustering? Could this potentially lead to unexpected behavior with RAFT elections or restoring the cluster state?

Issue 2: Missing Routes in Cluster Secret

In the cluster secret for routes, I observe only one route instead of two for each pod. The configured routes look like this:

routes = [
  nats://nats_cluster:password@nats:6222
]

According to the JetStream clustering documentation, I expected to see routes for each pod in the cluster. This seems like a misconfiguration. Could this contribute to the behavior described in Issue 1?

Questions:

1. Is the inconsistent connection reporting across pods expected, or could it indicate a deeper issue with the RAFT leader election or cluster synchronization?
2. Should the routes configuration in the cluster secret include all routes for every pod? If so, how can I ensure this is correctly set using the Bitnami Helm chart?

[MauriceVanVeen]

This is an issue with the Bitnami Helm chart:
bitnami/charts#30124

The complete set of pods should be set as part of the routes section in the configuration.

[wallyqs]

This is fixed in the bitnami charts now: bitnami/charts#30124