Grafana is not showing data from object store(s3) of thanos

[sreekanthm]

Hi,

Our observability cluster grafana is not pulling historical data from s3 bucket(object store), not seeing any error and data is already in S3 bucket and we are using bitnami thanos in our cluster. Here is our thanos configuration. let me know If we are missing

objstoreConfig: |-
type: s3
config:
bucket: xxxxx-xxxxx-xxxx-us-east-1
endpoint: s3.us-east-1.amazonaws.com:443
access_key: "XXXXXXXXXXX"
secret_key: "XXXXXXXXXXX"
query:
dnsDiscovery:
enabled: false

stores:
- XXXXXXXXXXXX.net:443
- YYYYYYYYYYYYY.net:443
- dadadaddadadadad.net:443

grpcTLS:
client:
secure: true
ca: |-
-----BEGIN CERTIFICATE-----
-------Certificate----------
-----END CERTIFICATE-----
bucketweb:
enabled: true
compactor:
enabled: true
storegateway:
enabled: true
ruler:
enabled: true
alertmanagers:
- http://prometheus-kube-prometheus-alertmanager:9093
config: |-
groups:
- name: "metamonitoring"
rules:
- alert: "PrometheusDown"
expr: absent(up{prometheus="monitoring/kube-prometheus"})
metrics:
enabled: true
serviceMonitor:
enabled: true

[yeya24]

Hi, we don't maintain the chart and I don't know if there is any thanos maintainer who are using the chart right now.
Can you give the thanos store and query component logs instead? That would be more helpful to debug your issue.

[sreekanthm]

Thanos store is as I mentioned in the above configuration(not mentioned the correct dns) and following this https://github.com/bitnami/charts/blob/master/bitnami/thanos/values.yaml

query logs

level=info ts=2022-01-17T23:55:54.776098604Z caller=client.go:55 msg="enabling client to server TLS"
level=info ts=2022-01-17T23:55:54.776437532Z caller=options.go:115 msg="TLS client using provided certificate pool"
ts=2022-01-17T23:55:54.778761261Z caller=log.go:168 level=debug msg="Lookback delta is zero, setting to default value" value=5m0s
level=info ts=2022-01-17T23:55:54.792966608Z caller=options.go:27 protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
level=info ts=2022-01-17T23:55:54.793815426Z caller=query.go:618 msg="starting query node"
level=info ts=2022-01-17T23:55:54.794209841Z caller=intrumentation.go:48 msg="changing probe status" status=ready
level=info ts=2022-01-17T23:55:54.798203916Z caller=intrumentation.go:60 msg="changing probe status" status=healthy
level=info ts=2022-01-17T23:55:54.79849836Z caller=http.go:63 service=http/server component=query msg="listening for requests and metrics" address=0.0.0.0:10902
level=info ts=2022-01-17T23:55:54.799574804Z caller=grpc.go:127 service=gRPC/server component=query msg="listening for serving gRPC" address=0.0.0.0:10901
ts=2022-01-17T23:55:54.799735548Z caller=log.go:168 service=http/server component=query level=info msg="TLS is disabled." http2=false
level=warn ts=2022-01-17T23:55:59.795919659Z caller=endpointset.go:512 component=endpointset msg="update of node failed" err="getting metadata: fallback fetching info from grafana-stg-thanos-sc.grafana.stg.aws.sncrcloud.net:443 after err: rpc error: code = DeadlineExceeded desc = context deadline exceeded: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=grafana-stg-thanos-sc.grafana.stg.aws.sncrcloud.net:443
level=info ts=2022-01-17T23:55:59.801114462Z caller=endpointset.go:360 component=endpointset msg="adding new sidecar with [storeAPI rulesAPI exemplarsAPI targetsAPI MetricMetadataAPI]" address=vnpt-maap-pac.vnpt-maaprcs.pac.usw-smh.sncrcloud.net:8443 extLset="{cluster="vnpt-maap-pac", prometheus="cattle-monitoring-system/rancher-monitoring-prometheus", prometheus_replica="prometheus-rancher-monitoring-prometheus-1"}"
level=info ts=2022-01-17T23:55:59.801308534Z caller=endpointset.go:360 component=endpointset msg="adding new sidecar with [storeAPI rulesAPI exemplarsAPI targetsAPI MetricMetadataAPI]" address=amp-maap-qa-thanos-sc.grafana.stg.aws.sncrcloud.net:443 extLset="{cluster="amp-maap-qa", prometheus="cattle-monitoring-system/rancher-monitoring-prometheus", prometheus_replica="prometheus-rancher-monitoring-prometheus-1"}"
level=info ts=2022-01-17T23:55:59.801398791Z caller=endpointset.go:360 component=endpointset msg="adding new sidecar with [storeAPI rulesAPI exemplarsAPI targetsAPI MetricMetadataAPI]" address=amp-maap-prod-thanos-sc.grafana.stg.aws.sncrcloud.net:443 extLset="{cluster="amp-maap-prod", prometheus="cattle-monitoring-system/rancher-monitoring-prometheus", prometheus_replica="prometheus-rancher-monitoring-prometheus-1"}"
level=info ts=2022-01-17T23:55:59.801521051Z caller=endpointset.go:360 component=endpointset msg="adding new sidecar with [storeAPI rulesAPI exemplarsAPI targetsAPI MetricMetadataAPI]" address=sip-sre-ptg-poc.grafana.stg.aws.sncrcloud.net:443 extLset="{cluster="sip-sre-ptg-poc", replica="sip-aws"}"
level=info ts=2022-01-17T23:55:59.801662942Z caller=endpointset.go:360 component=endpointset msg="adding new sidecar with [storeAPI rulesAPI exemplarsAPI targetsAPI MetricMetadataAPI]" address=cloud-sncr-pc-dev-thanos-sc.grafana.stg.aws.sncrcloud.net:443 extLset="{cluster="cloud-sncr-pc-dev", prometheus="cattle-monitoring-system/rancher-monitoring-prometheus", prometheus_replica="prometheus-rancher-monitoring-prometheus-0"}"
level=info ts=2022-01-17T23:55:59.801823121Z caller=endpointset.go:360 component=endpointset msg="adding new sidecar with [storeAPI rulesAPI exemplarsAPI targetsAPI MetricMetadataAPI]" address=cloud-assurant-pc-stg.grafana.stg.aws.sncrcloud.net:443 extLset="{cluster="cloud-assurant-pc-stg", prometheus="cattle-monitoring-system/rancher-monitoring-prometheus", prometheus_replica="prometheus-rancher-monitoring-prometheus-1"}"
level=info ts=2022-01-17T23:55:59.801923311Z caller=endpointset.go:360 component=endpointset msg="adding new sidecar with [storeAPI rulesAPI exemplarsAPI targetsAPI MetricMetadataAPI]" address=cloud-att-pc-stg.grafana.stg.aws.sncrcloud.net:443 extLset="{cluster="cloud-att-pc-stg", prometheus="cattle-monitoring-system/rancher-monitoring-prometheus", prometheus_replica="prometheus-rancher-monitoring-prometheus-1"}"
level=info ts=2022-01-17T23:55:59.802042026Z caller=endpointset.go:360 component=endpointset msg="adding new sidecar with [storeAPI rulesAPI exemplarsAPI targetsAPI MetricMetadataAPI]" address=thanos-digital-poc.dev.fa.oci.sncrcloud.net:443 extLset="{cluster="fa-digital-poc", replica="poc"}"
level=info ts=2022-01-17T23:55:59.802127027Z caller=endpointset.go:360 component=endpointset msg="adding new sidecar with [storeAPI rulesAPI exemplarsAPI targetsAPI MetricMetadataAPI]" address=cloud-vzw-nab-stg-dc1.vzw-nab.stg01.us-west.sncrcloud.net:443 extLset="{cluster="cloud-vzw-nab-stg-dc1", prometheus="cattle-monitoring-system/rancher-monitoring-prometheus", prometheus_replica="prometheus-rancher-monitoring-prometheus-0"}"
level=info ts=2022-01-17T23:55:59.802159119Z caller=endpointset.go:360 component=endpointset msg="adding new sidecar with [storeAPI rulesAPI exemplarsAPI targetsAPI MetricMetadataAPI]" address=gis-aqua-uat-eks-thanos-sc.security.uat.cloud.synchronoss.net:443 extLset="{cluster="gis-aqua-uat-eks", prometheus="cattle-monitoring-system/rancher-monitoring-prometheus", prometheus_replica="prometheus-rancher-monitoring-prometheus-1"}"
level=info ts=2022-01-17T23:55:59.802195161Z caller=endpointset.go:360 component=endpointset msg="adding new sidecar with [storeAPI rulesAPI exemplarsAPI targetsAPI MetricMetadataAPI]" address=cloud-tracfone-pc-stg.grafana.stg.aws.sncrcloud.net:443 extLset="{cluster="cloud-tracfone-pc-stg", prometheus="cattle-monitoring-system/rancher-monitoring-prometheus", prometheus_replica="prometheus-rancher-monitoring-prometheus-0"}"
level=warn ts=2022-01-17T23:56:04.80344609Z caller=endpointset.go:512 component=endpointset msg="update of node failed" err="getting metadata: fallback fetching info from grafana-stg-thanos-sc.grafana.stg.aws.sncrcloud.net:443 after err: rpc error: code = DeadlineExceeded desc = context deadline exceeded: rpc error: code = DeadlineExceeded desc = context deadline exceeded" address=grafana-stg-thanos-sc.grafana.stg.aws.sncrcloud.net:443

[yeya24]

Would be useful if you can provide the manifests of query and thanos store, too. K8s yaml not helm.