Unexpected 1 min timeout with Grafana #3036

andreivaloshyn · 2022-09-23T15:40:13Z

andreivaloshyn
Sep 23, 2022

Hello!
Some queries a failed because of timeout=1min
Error from Grafana:

status:504
statusText:""
data:Object
message:"<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
"
error:""
response:"<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
"
config:Object
url:"api/ds/query"
method:"POST"
data:Object
requestId:"Q160"
retry:0
headers:Object
hideFromInspector:false
message:"<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
"

Helm config with most important params(resource limits and nodeSelector are deleted)

mimir:
	  config: |
	    multitenancy_enabled: false
	
	    limits:
	      max_global_series_per_user: 10000000
	      max_global_series_per_metric: 10000000
	      max_label_names_per_series: 300
	      ruler_max_rules_per_rule_group: 1000
	      ingestion_rate: 150000
	      ruler_max_rule_groups_per_tenant: 100
	      max_query_parallelism: 140
	
	    activity_tracker:
	      filepath: /data/metrics-activity.log
	
	    frontend_worker:
	      frontend_address: {{ template "mimir.fullname" . }}-query-frontend-headless.{{ .Release.Namespace }}.svc:{{ include "mimir.serverGrpcListenPort" . }}
	      dns_lookup_duration: 1s
	      grpc_client_config:
	        backoff_on_ratelimits: true
	
	    alertmanager_storage:
	      backend: s3
	      s3:
	        bucket_name: cluster1-grafana-alertmanager-store
	        endpoint: s3.amazonaws.com
	        region: eu-west-1
	        insecure: false
	
	    alertmanager:
	      data_dir: /data
	      enable_api: true
	      external_url: /alertmanager
	
	    ruler:
	      alertmanager_url: dnssrvnoa+http://_http-metrics._tcp.{{ template "mimir.fullname" . }}-alertmanager-headless.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}/alertmanager
	      enable_api: true
	      rule_path: /data
	      ring:
	        heartbeat_timeout: 2m
	
	    querier:
	      timeout: 5m
	      default_evaluation_interval: 120s
	
	    store_gateway:
	      sharding_ring:
	        heartbeat_timeout: 2m
	
	    ruler_storage:
	      backend: s3
	      s3:
	        bucket_name: cluster1-grafana-alertmanager-store
	        endpoint: s3.amazonaws.com
	        region: eu-west-1
	        insecure: false
	
	    server:
	      grpc_server_max_recv_msg_size: 104857600
	      grpc_server_max_send_msg_size: 104857600
	      grpc_server_max_concurrent_streams: 1000
	      log_format: json
	      log_level: debug
	      http_server_read_timeout: 360s
	      http_server_write_timeout: 120s
	      graceful_shutdown_timeout: 120s
	
	    frontend:
	      log_queries_longer_than: 10s
	      align_queries_with_step: true
	      results_cache:
	        backend: memcached
	        memcached:
	          addresses: monitoring-memcached-grafana.cluster1.domain:11211
	      cache_results: true
	
	    compactor:
	      data_dir: "/data"
	
	    ingester:
	      instance_limits:
	        max_ingestion_rate: 0
	        max_series: 0
	      ring:
	        final_sleep: 0s
	        num_tokens: 512
	        unregister_on_shutdown: false
	        tokens_file_path: /data/token
	
	    ingester_client:
	      grpc_client_config:
	        max_recv_msg_size: 104857600
	        max_send_msg_size: 104857600
	
	    runtime_config:
	      file: /var/{{ include "mimir.name" . }}/runtime.yaml
	
	    memberlist:
	      abort_if_cluster_join_fails: false
	      compression_enabled: false
	      join_members:
	      - {{ include "mimir.name" . }}-gossip-ring
	
	    # This configures how the store-gateway synchronizes blocks stored in the bucket. It uses Minio by default for getting started (configured via flags) but this should be changed for production deployments.
	    blocks_storage:
	      backend: s3
	      tsdb:
	        dir: /data/tsdb
	        retention_period: 24h
	      bucket_store:
	        sync_dir: /data/tsdb-sync
	        chunks_cache:
	          backend: memcached
	          memcached:
	            addresses: monitoring-memcached-grafana.cluster1.domain:11211
	        metadata_cache:
	          backend: memcached
	          memcached:
	            addresses: monitoring-memcached-grafana.cluster1.domain:11211
	          metafile_attributes_ttl: 720h
	          block_index_attributes_ttl: 720h
	        index_cache:
	          backend: memcached
	          memcached:
	            addresses: monitoring-memcached-grafana.cluster1.domain:11211
	        bucket_index:
	          enabled: true
	
	      s3:
	        bucket_name: cluster1-grafana-mimir-store
	        endpoint: s3.amazonaws.com
	        region: eu-west-1
	        insecure: false
	        sse:
	          type: SSE-S3
	        http:
	          expect_continue_timeout: 360s
	          max_idle_connections: 500
	rbac:
	  pspEnabled: false
	
	serviceMonitor:
	  enabled: true
		
	alertmanager:
	  enabled: true
	  replicas: 1
	  persistentVolume:
	    size: 10Gi
	    storageClass: aws-ebs-csi-retain
	
	distributor:
	  replicas: 3

	ingester:
	  replicas: 3
	  persistentVolume:
	    size: 70Gi
	    storageClass: aws-ebs-csi-retain
	
	overrides_exporter:
	  enabled: false
	  replicas: 1
	
	ruler:
	  enabled: true
	  replicas: 1
	
	querier:
	  replicas: 3
	
	query_frontend:
	  replicas: 3
	
	  nodeSelector:
	    type: system
	    kubernetes.io/os: linux
	    kubernetes.io/arch: amd64
	
	store_gateway:
	  replicas: 1
	  persistentVolume:
	    size: 30Gi
	    storageClass: aws-ebs-csi-retain
	
	compactor:
	  replicas: 1
	
	  persistentVolume:
	    size: 30Gi
	    storageClass: aws-ebs-csi-retain
	
	nginx:
	  nginxConfig:
	    httpSnippet: |-
	      proxy_read_timeout 600;
	      proxy_connect_timeout 600;
	      proxy_send_timeout 600;
	  enabled: true
	  replicas: 1
	  ingress:
	    enabled: true
	    tls:
	      - hosts: [ mimir.cluster1.domain ]
	        secretName: mimir.cluster1.domain
	    hosts:
	      - host: mimir.cluster1.domain
	        paths:
	          - path: /
	            pathType: ImplementationSpecific
	    annotations:
	      external-dns/public: 'false'
	      cert-manager.io/cluster-issuer: cluster-issuer-lets-encrypt
	      external-dns.alpha.kubernetes.io/hostname: mimir.cluster1.domain.
	    ingressClassName: nginx-private
	metaMonitoring:
	  serviceMonitor:
	    interval: 15s
	    enabled: true
	    labels:
	      instance: cluster1

Helm chart is mimir-distributed-3.0.0
datasource endpoint config:

- name: Mimir
  jsonData:
    timeInterval: 15s
    queryTimeout: 360s
    timeout: 360
  type: prometheus
  access: proxy
  isDefault: false
  url: http://mimir-nginx.mimir.svc.cluster.local:80/prometheus
  editable: true

Screenshot attached
VictoriaMetrics have no this timeout problem, so not a browser problem I guess

56quarters · 2022-09-23T15:55:56Z

56quarters
Sep 23, 2022
Maintainer

Could you please include some logs from the Mimir cluster in question and verify that all components are running?

12 replies

andreivaloshyn Sep 28, 2022
Author

Nope. I also tried to use query_frontend k8s service endpoint directly and got the same 1 min timeout. So it's not nginx.

dimitarvdimitrov Sep 28, 2022
Maintainer

i would check the connection between the host where grafana runs to the host where the query frontend works.

seems like they are both in kubernetes.

kubectl exec -it grafana sh
telnet mimir-query-frontend.mimir.svc.cluster.local 8080

Since it's not the nginx, then do you see any logs in the query_frontend when these timeouts occur?

andreivaloshyn Sep 29, 2022
Author

With debug no logs related to this timeout.
Also grafana error message says

<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>

Feels like we have nginx somewhere else.

carlosjgp Sep 30, 2022

Grafana Mimir uses Nginx in its gateway component to route all the requests to the different components

504 is normally caused because there are no healthy instances in the upstream... maybe a component is being OOMKilled?

andreivaloshyn Sep 30, 2022
Author

@carlosjgp nginx was configured with 600s timeout. And no resource were struggling that time because of CPU or Memory limits.
So pretty sure this is not related to OOMKilled.
But it can be related to store-gateway inability to process requests. With a single store-gateway pod I faced this timeout much more frequently comparing with 3 pods of store-gateway.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Unexpected 1 min timeout with Grafana #3036

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment 12 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Unexpected 1 min timeout with Grafana #3036

Uh oh!

Uh oh!

andreivaloshyn Sep 23, 2022

Replies: 1 comment · 12 replies

Uh oh!

56quarters Sep 23, 2022 Maintainer

Uh oh!

andreivaloshyn Sep 28, 2022 Author

Uh oh!

dimitarvdimitrov Sep 28, 2022 Maintainer

Uh oh!

andreivaloshyn Sep 29, 2022 Author

Uh oh!

carlosjgp Sep 30, 2022

Uh oh!

andreivaloshyn Sep 30, 2022 Author

andreivaloshyn
Sep 23, 2022

Replies: 1 comment 12 replies

56quarters
Sep 23, 2022
Maintainer

andreivaloshyn Sep 28, 2022
Author

dimitarvdimitrov Sep 28, 2022
Maintainer

andreivaloshyn Sep 29, 2022
Author

andreivaloshyn Sep 30, 2022
Author