Not able to understand why is thanos is not deleting really old data?

[dharapvj]

Hi, I have a thanos sidecar and standalone thanos setup.

Thanos has below retention setting:

    compact:
      retention:
        resolutionRaw: 2d
        resolution5m: 7d
        resolution1h: 60d

pometheus is started with below params:

            --storage.tsdb.retention.time=5d 
            --storage.tsdb.min-block-duration=2h 
            --storage.tsdb.max-block-duration=2h 

But as can be seen from screenshots... sidecar is not clearing data all the way back to Sept 10th 2021 (almost 130 days old)

I don't understand why

As we can see below.. the data is stored for so long is provided by sidecar

How can clear up old blocks so that I only retain 60 days worth of data?

[fktkrt]

Hey, could you please share your Compactor's config/flags?

[dharapvj]

thanos is started with below args

compact --data-dir=/var/compact-data --objstore.config-file=/etc/thanos/objstore.yaml --retention.resolution-raw=2d --retention.resolution-5m=7d --retention.resolution-1h=60d --wait

objstore.yaml has below contents:

config:
  access_key: <my_access_key>
  bucket: thanos
  endpoint: minio.minio.svc.cluster.local:9000
  insecure: true
  secret_key: <my_secret_key>
type: S3

Hope that helps.