MimirCompactorHasNotSuccessfullyRunCompaction | checksum mismatch

[JustNZ]

Describe the bug

Mimir reports the following alarm MimirCompactorHasNotSuccessfullyRunCompaction

in the logs of mimir I see the following:
ts=2024-03-12T13:48:32.164077229Z caller=compactor.go:652 level=error component=compactor msg="failed to compact user blocks" user=test err="compaction: group 0@17241709254077376921-merge--1695859200000-1695945600000: compact blocks [/var/lib/mimir/data/data-compactor/compact/0@17241709254077376921-merge--1695859200000-1695945600000/01HBFDCC69FYA7DY6Q7N05TYXD /var/lib/mimir/data/data-compactor/compact/0@17241709254077376921-merge--1695859200000-1695945600000/01HBFCHQSCV2KY6HSC1PM6FRTM]: populate block: chunk iter: cannot populate chunk 42949673709 from block 01HBFCHQSCV2KY6HSC1PM6FRTM: checksum mismatch expected:3b12049, actual:f421572c"

To Reproduce

Steps to reproduce the behavior:

I tried to unify the data from two mimir clusters into one and copied blocks to a single s3 bucket.
This seems to cause the issue. see: #7563

Expected behavior

Resolve issue of checksum mismatch

Environment

• Infrastructure: 4 Server cluster single binary
• Deployment tool: ansible

Additional Context

[pstibrany]

Block is now corrupted, and cannot be compacted anymore. You can use markblocks tool from Mimir repository to mark block for non-compaction. If you are running Mimir 2.11, it has a feature to mark corrupted blocks for non-compaction automatically.

* [ENHANCEMENT] Compactor: Mark corrupted blocks for no-compaction to avoid blocking compactor future runs. #6588