Out-of-Order Sample Errors with Alloy Remote Write to Mimir

[girish-ms]

Description

We are experiencing frequent out-of-order sample rejections when using Alloy to remote_write to Mimir. The issue occurs across multiple metric series, and the affected samples have timestamps that are significantly different from each other.

Alloy Logs:(example)

ts=2025-03-07T13:42:26.330800121Z level=error msg="non-recoverable error" component_path=/ component_id=prometheus.remote_write.default subcomponent=rw remote_name=985df6 url=http://mimir-distributor.mimir.svc.cluster.local:8080/api/v1/push failedSampleCount=38 failedHistogramCount=0 failedExemplarCount=0 err="server returned HTTP status 400 Bad Request: send data to ingesters: failed pushing to ingester mimir-ingester-0: user=anonymous: the sample has been rejected because another sample with a more recent timestamp has already been ingested and out-of-order samples are not allowed (err-mimir-sample-out-of-order). The affected sample has timestamp 2025-03-07T13:04:02.887Z and is from series go_memory_classes_heap_objects_bytes{instance=\"ip-xxxxxxxx.us-west-2.compute.internal\", job=\"integrations/kubernetes/kubelet\"} (sampled 1/10)"

Distributor Logs:(example)

ts=2025-03-07T13:42:07.060631562Z caller=push.go:221 level=error user=anonymous msg="detected an error while ingesting Prometheus remote-write request (the request may have been partially ingested)" httpCode=400 err="send data to ingesters: failed pushing to ingester mimir-ingester-3: user=anonymous: the sample has been rejected because another sample with a more recent timestamp has already been ingested and out-of-order samples are not allowed (err-mimir-sample-out-of-order). The affected sample has timestamp 2025-03-07T13:06:52.418Z and is from series container_cpu_load_average_10s{id=\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fe26629_5479_483e_8e29_ec171951e388.slice/cri-containerd-b192c4e181597b618062b91cd71e7ca05ea9298cb7e0773b9ab5d342372841e8.scope\", image=\"xxxxxx.dkr.ecr.us-west-2.amazonaws.com/eks/pause:3.5\", instance=\"ip-xxxx.us-west-2.compute.internal\", job=\"integrations/kubernetes/cadvisor\", name=\"b192c4e181597b618062b91cd71e7ca05ea9298cb7e0773b9ab5d342372841e8\", namespace=\"kube-system\", pod=\"node-xxxxx\"} (sampled 1/10)"

Observations

• The timestamps of rejected samples are several minutes apart, making it unclear why they are being classified as out-of-order.
• Logs show errors across Alloy, Mimir Distributor, and Mimir Ingester, indicating the problem exists throughout the ingestion pipeline.
• The issue is not isolated to a single job or namespace.

Questions:

• What debugging steps can help determine whether the issue originates from Alloy, Mimir Distributor, or Mimir Ingester?
• Are there any best practices or mitigation strategies to prevent out-of-order sample rejections in this setup?

Tagging @pracucci for quick resolution. Any insights would be greatly appreciated!

[shazinahmed]

I am facing the exact same issue with chart mimir-distributed-5.7.0. I have ensured

1. The clocks are in sync in all the nodes
2. There are no other scrapers running
3. Disabled zone aware ingestion and storage (wasn't sure if this was messing things up).

I am unable to get any metrics because of this error.

@girish-ms Where you able to resolve this issue?

[pribeiro-s2]

I have also been debugging this for the past few days to no avail. The behavior we are noticing are gaps in metrics in Mimir, or they arrive with significant delay, and the only errors/warnings we see in Mimir are the same as the one reported in this ticket. For reference we:

1. Have HA Deduplication enabled
2. Are running Mimir with replication factor of 3
3. Also have disabled zone awareness
4. Have upgraded to mimir-distributed-5.7.0, but were observing the similar issue with 5.4.0
5. Have Ingress Nginx pointing to Mimir Gateway
   • Initially we weren't using Gateway, but experimented to see if we were missing any important config

We have a very similar setup with Cortex where we don't see any of these issues. We are essentially deploying Mimir in parallel so we can eventually cutover, and are at a point where Mimir is even overprovisioned in comparison to Cortex, which makes it very unlikely for this to be a resource usage issue. We have deployed new Prometheis running v2.54.1 whilst the old ones (which are pushing to Cortex) are running v2.32.1.

We were able to mitigate this issue by relaxing the queue_config in our Prometheis remote write blocks, and have gone from 30m/1h/2h gaps to just some minutes, but it is still something I'd really like to fix.