Loki Ingesters and Mimir Ingesters Sharing a Ring?!

[alextricity25]

I woke up this morning to the strangest state of my telemetry cluster.

I install mimir using the mimir-distributed helm chart, and the loki helm chart in microservies mode on a kubernetes cluster. Both are on their own namespace. However, this morning the Mimir queriers were throwing this error:
error msg="cancelling all outstanding requests because quorum cannot be reached"

I checked the mimir ring by port-forwarding the mimir ingester pod and going to https://localhost:8080/ingester/ring, and found that the loki ingesters were on the same ring as the mimir ingesters?! Has anyone seen this before? Surely this isn't normal, right?

[pracucci]

K8S pods suffers an issue that we call "Fast IPs recycling". The problem is that the IP of a terminating Loki pod may be quickly reused for a starting Mimir pod (viceversa can happen too). If there's not enough delay between when a IP is released and that IP is reused (recycled) for another pod, other Loki pods may connect to that IP believing it's still a Loki pod (when it's actually a Mimir pod now), and this can cause the two memberlist clusters to join.

How to protect from this? The solution is to configure the following in Mimir to enable memberlist cluster "verification" (Loki has similar settings too):

# (advanced) The cluster label is an optional string to include in outbound
# packets and gossip streams. Other members in the memberlist cluster will
# discard any message whose label doesn't match the configured one, unless the
# 'cluster-label-verification-disabled' configuration option is set to true.
# CLI flag: -memberlist.cluster-label
[cluster_label: <string> | default = ""]

# (advanced) When true, memberlist doesn't verify that inbound packets and
# gossip streams have the cluster label matching the configured one. This
# verification should be disabled while rolling out the change to the configured
# cluster label in a live memberlist cluster.
# CLI flag: -memberlist.cluster-label-verification-disabled
[cluster_label_verification_disabled: <boolean> | default = false]

To rollout this change with no downtime, you will have to change config in a couple of steps. Please refer to this doc:
https://grafana.com/docs/helm-charts/mimir-distributed/latest/migration-guides/migrate-enable-cluster-label-verification/

[alextricity25]

Makes perfect sense. I'll configure those values. Thank you so much!