store-gateway issue : busy with index header forming and lazy index-header loading when new long range queries came after replacement of a new empty disk

[madhu-reddy-peram]

We did a deployment of store gateways with a change in IOPS in PremiumV2 azure disk zone by zone(This disk deployment clears all data from disk and new empty disk is attached)

During store gateway rollout we saw each store gateway downloaded bucket index for the blocks first and then built index-headers on to the disk at path /data/tsdb/fake. The below first screenshot shows it took more than an hour on 26th Feb.( 12:45 UTC to 14:00 UTC). During this time pod is not ready ( readiness probe failed as expected). Refer below screenshot-

Then when new queries with long time range started coming next day morning(05:00 AM onwards) , we saw store gateways going into not serving mode ( even prometheus not able to scrape metrics from /metrics endpoint) with below logs in store gateways. This kept happening for the intervals shown below -

Log lines during this time -
loading sparse index-header from full index-header
writing sparse index-header to disk
wrote sparse index-header to disk

We are assuming this started happening due to new long time range queries started coming and then lazy loading started happening.

During above process happening in store gateway pods on both occasions, we saw the throughput of the disk has reached maximum we set - Refer below screenshot you can see the second occurrence on early morning of 27th Feb.

Also CPU usage reduced during this time, but memory keep increasing but thats expected I guess.

Could you please help here to understand below during this 2nd instance of lazy loading on 27th Morning -

• why the store gateway pods are not responding to traffic when lazy loading process started? is this a expected behaviour?
• Why even /metrics endpoint is not available. This we confirmed by looking at metrics in grafana?
• Does throughput limit on the disk has any adverse effect on traffic serving during this time?

[dimitarvdimitrov]

• why the store gateway pods are not responding to traffic when lazy loading process started? is this a expected behaviour?

the store-gateway needs to load the index header from disk or construct it from the bucket if it's not on disk before being able to serve queries. So that's exected behaviour

• Why even /metrics endpoint is not available. This we confirmed by looking at metrics in grafana?

The metrics endpoint should be available, nothing obvious comes to mind. Some vague direction is that you said the disk throughput was reached - if the disk is network-attached, then this can also max-out the bandwidth of the node and end up throttling both disk access and network access (thus preventing /metrics from receiving the request/response)

• Does throughput limit on the disk has any adverse effect on traffic serving during this time?

if it slows down loading of index headers, then yes.

Consider disabling lazy loading. This will increase startup time, but shouldn't impact queries. -blocks-storage.bucket-store.index-header-lazy-loading-enabled=false

[madhu-reddy-peram]

@dimitarvdimitrov - Thanks for your reply! We will take a look at the suggested approach on lazy loading.