422 Unprocessable Entity, body: \"block time range crosses boundary of configured compactor time range (1d)

[mifinn85]

Hi,

We are trying to upload TSDB blocks from Prometheus to Mimir but it keeps failing on time range set to 1d, older block files are not being uploaded.

Where exactly we need to change this setting from 1d to a different value.

Helm Release:

  chart: mimir-distributed

  version: 5.6.0-weekly.316

Error:

/prometheus $ ./mimirtool backfill --address=http://mimir-compactor.monitoring.svc.cluster.local:8080 --id=anonymous /prometheus/01J9HCHX3VV8W28445J8G7DGEC
INFO[0000] Backfilling blocks=/prometheus/01J9HCHX3VV8W28445J8G7DGEC user=anonymous
INFO[0000] making request to start block upload block=01J9HCHX3VV8W28445J8G7DGEC file=meta.json path=/prometheus/01J9HCHX3VV8W28445J8G7DGEC
ERRO[0000] response body="block time range crosses boundary of configured compactor time range (1d)\n" status="422 Unprocessable Entity"
ERRO[0000] failed uploading block error="request to start block upload failed: POST request to http://mimir-compactor.monitoring.svc.cluster.local:8080/api/v1/upload/block
/01J9HCHX3VV8W28445J8G7DGEC/start failed: server returned HTTP status: 422 Unprocessable Entity, body: "block time range crosses boundary of configured compactor time range (1d)\n"" path=/pro
metheus/01J9HCHX3VV8W28445J8G7DGEC
INFO[0000] finished uploading blocks already_exists=0 failed=1 succeeded=0
mimirtool: error: blocks failed to upload 1 block(s), try --help

[mifinn85]

Would really appreciate some help here please.

[andyasp]

Hello,

Where exactly we need to change this setting from 1d to a different value.

I would not suggest changing the -compactor.block-ranges setting. It's best to leave that at the default.

ERRO[0000] response body="block time range crosses boundary of configured compactor time range (1d)\n" status="422 Unprocessable Entity"

The error you are encountering is due to an uploaded block not aligning to the time boundaries that are expected by Mimir. Mimir, by default, expects at most 24-hour blocks aligned by day. For example, if you try to upload a block that spans noon of one day to noon of the following day you would hit this error. This validation was added in #9524. Ensuring this alignment helps avoid encountering performance issues later on in the query path.

The solution is to use a tool in the Mimir repository, splitblocks, to split the original block into multiple blocks that have an expected alignment. Then those blocks can be uploaded instead. It would certainly be nice if the backfill did this for you, but that hasn't been implemented.

[azhar1038]

I would not suggest changing the -compactor.block-ranges setting. It's best to leave that at the default.

I have a block compacted by prometheus which is of 21 days duration, I tried changing block-ranges to 30 days (experimenting) but still I got the same error.

I can see in the code

	if blockMaxTime.After(blockMinTime.Truncate(maxRange).Add(maxRange)) {
		return httpError{
			message:    fmt.Sprintf("block time range crosses boundary of configured compactor time range (%v)", model.Duration(maxRange)),
			statusCode: http.StatusUnprocessableEntity,
		}
	}

I have never used go before but this condition looks very weird. I tried running a sample program with following values:

blockMinTime = 1715625000000 (2024-05-13 18:30:00 +0000 UTC)
blockMaxTime = 1716316200001 (2024-05-21 18:30:00.001 +0000 UTC)
maxRange = 30 * 24 * time.Hour (30 days)
blockMinTime.Truncate(maxRange).Add(maxRange) gives 2024-05-15 00:00:00 +0000 UTC

which is not even 2 days!
Is this intentional or bug?

I believe it should be: blockMaxTime.After(blockMinTime.Truncate(time.Hour).Add(maxRange))

[andyasp]

Is this intentional or bug?

Hi @azhar1038, it's intentional. Let's break down blockMaxTime.After(blockMinTime.Truncate(maxRange).Add(maxRange)) to see why.

The docs for Truncate are here: https://pkg.go.dev/time#Time.Truncate

Truncate returns the result of rounding t down to a multiple of d (since the zero time).

maxRange is commonly 1 day, so let's stick to that to orient ourselves. Given that, blockMinTime.Truncate(maxRange) is finding the start of the day corresponding to blockMinTime. Then adding maxRange to that finds the end of the day. If blockMaxTime is more than that we know the block is crossing a day boundary. One way to think about this is that blocks have to fit into "slots". Those "slots" are determined by the maximum range and we want all blocks to fall within them and not across them.

Now let's go to your example. Where I think you're getting confused is that you are thinking your blockMinTime is near the start of the "slot" determined by the maxRange of 30 days. It isn't, it's falling near the end of it.

Here's a go playground to try to make that clearer: https://go.dev/play/p/57M7-jb-lk5:

package main

import (
	"fmt"
	"time"
)

func main() {
	minTime := time.UnixMilli(1715625000000)
	fmt.Println("minTime", minTime)
	maxRange := 30 * 24 * time.Hour
	truncated := minTime.Truncate(maxRange)
	fmt.Println("truncated", truncated)
	end := truncated.Add(maxRange)
	fmt.Println("end", end)
}

Gives:

minTime 2024-05-13 18:30:00 +0000 UTC
truncated 2024-04-15 00:00:00 +0000 UTC
end 2024-05-15 00:00:00 +0000 UTC

Hope that helps.

[azhar1038]

Okay, thanks for the explanation. Then the only possible ways are:

• Set some very huge block-range like 4 months, but then also not sure, if all blocks will pass the check
• Use Splitblocks (Probably only option)