Compactor no longer compacts after ingester issues

[braamvh-nintex]

Hi all. I have a situation where my compactors no longer compact properly. I get a lot of messages like these:

{"caller":"compactor.go:652","component":"compactor","err":"compaction: group 0@17241709254077376921-merge--1728475200000-1728482400000: gather index issues for block /data/compact/0@17241709254077376921-merge--1728475200000-1728482400000/01J9SBW4BEK40A5921XE632KWC: get all postings: decode postings: invalid size","level":"error","msg":"failed to compact user blocks","ts":"2024-10-23T12:09:56.936917013Z","user":"<removed>"}

The ingesters were incorrectly configured (without disks), and when I added that in, they started corrupting blocks. I finally managed to get a stable configuration for the ingesters, and it is possible they uploaded some corrupted data to the block store.

I went and deleted all the blocks (01J9SBW4BEK40A5921XE632KWC in the message above) and restarted the compactors, but it came up with a whole new list on the subsequent run. Am I on the right track, or is this not the correct approach? Is there a way to identify the corrupted blocks here? I am using an Azure storage container.