streaming chunk from store gateway failing

[kadhamecha-conga]

hi team,

I have enable below config to reduce querier memory usage , but on large range query is failing with below error:

execution: attempted to read series at index 16757 from stream, but the stream has failed: rpc error: code = Unavailable desc = closing transport due to: connection error: desc = "error reading from server: read tcp x.x.x.x:52180->x.x..x.x:9095: read: connection reset by peer", received prior goaway: code: NO_ERROR, debug data: "max_age"

---

below two values i have added as querier config:
prefer_streaming_chunks_from_store_gateways: true
streaming_chunks_per_store_gateway_series_buffer_size: 512

can you please help me to undersatnd where the issue is ?
what should be the value set for streaming_chunks_per_store_gateway_series_buffer_size , if query range is higher ?

thanks.

[charleskorn]

That error message makes it sound as if the store-gateway the querier was reading from has crashed. Could you please check for any issues in the store-gateway logs around the same time?

[kadhamecha-conga]

hi,

there is no such error, but just below warnings in same sg logs:

caller=memcached_client.go:335 level=warn name=chunks-cache msg="failed to fetch items from memcached" numKeys=45 firstKey=subrange:anonymous/01HDMT2JB20N7P7CHPA88D6KNN/chunks/000002:85744000:85760000 err="memcache: connect timeout to 10.7.27.130:11211"

i'm running with below caching component:
chunk: 4
index: 2
metadata: 1
result: 2

i'm trying to use stream to reduce the querier memory usage and start facing this issue.
resources usage is under threshold.

let me know, if you need more info.

thanks.

[charleskorn]

The error message you posted contains a string like error reading from server: read tcp x.x.x.x:52180->y.y.y.y:9095: read: connection reset by peer. What component does the second address (y.y.y.y:9095) refer to? I'd check if that is crashing or experiencing any network-related issues.

[charleskorn]

current values of streaming_chunks_per_store_gateway_series_buffer_size 10000.

This is very high. The default value of 256 should be fine.

[kadhamecha-conga]

if i remove streaming option for sg query is failing with consistency-check-failed
and if i added stream large query failed with
execution: attempted to read series at index 16757 from stream, but the stream has failed

[charleskorn]

Could you please share your entire configuration file, with any sensitive values removed?

[kadhamecha-conga]

PFA config.

`

activity_tracker:
  filepath: /active-query-tracker/activity.log
alertmanager:
  data_dir: /data
  enable_api: true
  external_url: /alertmanager
  fallback_config_file: /configs/alertmanager_fallback_config.yaml
blocks_storage:
  backend: s3
  bucket_store:
    chunks_cache:
      backend: memcached
      memcached:
        addresses: dns+mimir-chunks-cache.dev.svc:11211
        max_async_buffer_size: 10000000
        max_async_concurrency: 100
        max_get_multi_batch_size: 1000
        max_get_multi_concurrency: 750
        max_idle_connections: 750
        max_item_size: 10485760
        timeout: 10s
    index_cache:
      backend: memcached
      memcached:
        addresses: dns+mimir-index-cache.dev.svc:11211
        max_async_buffer_size: 10000000
        max_async_concurrency: 100
        max_get_multi_batch_size: 1000
        max_get_multi_concurrency: 750
        max_idle_connections: 750
        max_item_size: 10485760
        timeout: 10s
    index_header_lazy_loading_enabled: true
    index_header_lazy_loading_idle_timeout: 1h
    max_chunk_pool_bytes: 12884901888
    metadata_cache:
      backend: memcached
      memcached:
        addresses: dns+mimir-metadata-cache.dev.svc:11211
        max_async_buffer_size: 10000000
        max_async_concurrency: 100
        max_get_multi_batch_size: 1000
        max_get_multi_concurrency: 750
        max_idle_connections: 750
        max_item_size: 10485760
        timeout: 10s
    sync_dir: /data/tsdb-sync
  s3:
    bucket_name: XXXXXX.
  tsdb:
    dir: /data/tsdb
    flush_blocks_on_shutdown: false
    head_compaction_interval: 15m
    wal_replay_concurrency: 3
common:
  storage:
    backend: s3
    s3:
      endpoint: s3.dualstack.xxx.com
      region: us-east-2
compactor:
  compaction_interval: 30m
  data_dir: /data
  deletion_delay: 2h
  first_level_compaction_wait_period: 25m
  max_closing_blocks_concurrency: 2
  max_opening_blocks_concurrency: 4
  sharding_ring:
    wait_stability_min_duration: 1m
  symbols_flushers_concurrency: 4
frontend:
  align_queries_with_step: true
  cache_results: true
  log_queries_longer_than: 5s
  parallelize_shardable_queries: true
  query_sharding_target_series_per_shard: 2500
  query_stats_enabled: true
  results_cache:
    backend: memcached
    memcached:
      addresses: dns+mimir-results-cache.dev.svc:11211
      max_async_buffer_size: 10000000
      max_async_concurrency: 100
      max_get_multi_batch_size: 1000
      max_get_multi_concurrency: 750
      max_idle_connections: 750
      max_item_size: 10485760
      timeout: 5s
  scheduler_address: mimir-query-scheduler-headless.dev.svc:9095
  split_queries_by_interval: 24h0m0s
frontend_worker:
  grpc_client_config:
    max_send_msg_size: 419430400
  scheduler_address: mimir-query-scheduler-headless.dev.svc:9095
ingester:
  ring:
    final_sleep: 0s
    num_tokens: 512
    replication_factor: 1
    tokens_file_path: /data/tokens
    unregister_on_shutdown: false
ingester_client:
  grpc_client_config:
    max_recv_msg_size: 104857600
    max_send_msg_size: 104857600
limits:
  cardinality_analysis_enabled: true
  compactor_blocks_retention_period: 30d
  ingestion_burst_size: 1000000
  ingestion_rate: 200000
  max_cache_freshness: 10m
  max_fetched_chunks_per_query: 10000000
  max_global_exemplars_per_user: 100000
  max_global_series_per_metric: 0
  max_global_series_per_user: 10000000
  max_label_names_per_series: 50
  max_query_parallelism: 960
  max_total_query_length: 30d
  query_sharding_max_sharded_queries: 0
memberlist:
  abort_if_cluster_join_fails: false
  compression_enabled: false
  join_members:
  - dns+mimir-gossip-ring.dev.svc.cluster.local:7946
querier:
  max_concurrent: 48
  timeout: 5m
    --># prefer_streaming_chunks_from_ingesters: true
    --># streaming_chunks_per_ingester_series_buffer_size: 10000
    --># prefer_streaming_chunks_from_store_gateways: true
    --># streaming_chunks_per_store_gateway_series_buffer_size: 10000
query_scheduler:
  max_outstanding_requests_per_tenant: 800
ruler:
  alertmanager_url: dnssrvnoa+http://_http-metrics._tcp.mimir-alertmanager-headless.dev.svc.cluster.local/alertmanager
  enable_api: true
  rule_path: /data
ruler_storage:
  cache:
    backend: memcached
    memcached:
      addresses: dns+mimir-metadata-cache.dev.svc:11211
      max_item_size: 10485760
runtime_config:
  file: /var/mimir/runtime.yaml
server:
  grpc_server_max_concurrent_streams: 1000
  grpc_server_max_connection_age: 2m
  grpc_server_max_connection_age_grace: 5m
  grpc_server_max_connection_idle: 1m
  grpc_server_max_recv_msg_size: 504857600
  grpc_server_max_send_msg_size: 504857600
  http_server_read_timeout: 5m0s
  http_server_write_timeout: 5m0s
store_gateway:
  sharding_ring:
    replication_factor: 2
    tokens_file_path: /data/tokens
    unregister_on_shutdown: false
    wait_stability_min_duration: 1m
usage_stats:
  installation_mode: helm
`

[charleskorn]

if i remove streaming option for sg query is failing with consistency-check-failed
and if i added stream large query failed with
execution: attempted to read series at index 16757 from stream, but the stream has failed

Could you please include the full error messages you see here? Both should include more information.

[kadhamecha-conga]

i'm running on 2.10.0.

[jnavarrof]

Hi @kadhamecha-conga. Hitting exactly the same error. Did you find the issue in your setup? Thanks

[mustafayildirim]

Any update on this? We're facing the same issue.
We're using helm

    chart: mimir-distributed
    repoURL: https://grafana.github.io/helm-charts
    targetRevision: 5.2.3

config

global:
  extraEnv:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: ""

rbac:
  create: false

mimir:
  structuredConfig:
    querier:
      max_concurrent: 10
      max_samples: 200000000
    frontend:
      parallelize_shardable_queries: true
    distributor:
      ha_tracker:
        enable_ha_tracker: true
        kvstore:
          store: consul
          consul:
            host: http://consul-server.consul:8500
    limits:
      accept_ha_samples: true
      # use different cluster label to support HA + sharding
      # see: https://github.com/grafana/mimir/issues/5217#issuecomment-1589266832
      ha_cluster_label: "ha_cluster"
      ingestion_rate: 10000000
      max_global_series_per_user: 0
      # Allow ingestion of out-of-order samples up to 60 minutes since the latest received sample for the series.
      out_of_order_time_window: 1h
      # default value of max_query_parallelism is 14, which should be multiplied by query_sharding_total_shards
      max_query_parallelism: 224
      query_sharding_total_shards: 16
      # 30 days -> now-30d gives 721h (daylight saving)
      max_total_query_length: 721h
      # 60 days
      max_query_lookback: 1440h
      # 60 days
      compactor_blocks_retention_period: 1440h
    blocks_storage:
      bucket_store:
        max_concurrent: 200
        block_sync_concurrency: 100
        bucket_index:
          max_stale_period: 6h
      backend: gcs
      storage_prefix: blocks
      gcs:
        bucket_name: gcs_mimir_bucket
    alertmanager_storage:
      backend: gcs
      storage_prefix: alertmanager
      gcs:
        bucket_name: gcs_mimir_bucket
    ruler_storage:
      backend: gcs
      storage_prefix: ruler
      gcs:
        bucket_name: gcs_mimir_bucket

minio:
  enabled: false

metaMonitoring:
  serviceMonitor:
    enabled: true
  prometheusRule:
    enabled: true
    mimirAlerts: true
    mimirRules: true

alertmanager:
  persistentVolume:
    enabled: true
  replicas: 2
  resources:
    limits:
      memory: 1.4Gi
    requests:
      cpu: 1
      memory: 1Gi
  statefulSet:
    enabled: true

compactor:
  replicas: 3
  persistentVolume:
    size: 50Gi
    storageClass: "premium-rwo"
  resources:
    limits:
      memory: 2.8Gi
    requests:
      cpu: 1
      memory: 2Gi

distributor:
  replicas: 48
  resources:
    limits:
      memory: 4Gi
    requests:
      cpu: 4
      memory: 4Gi

ingester:
  persistentVolume:
    size: 50Gi
  replicas: 48
  resources:
    limits:
      memory: 24Gi
    requests:
      cpu: 2
      memory: 24Gi


chunks-cache:
  enabled: true
  replicas: 12

index-cache:
  enabled: true
  replicas: 12

metadata-cache:
  enabled: true

results-cache:
  enabled: true
  replicas: 3
  allocatedMemory: 1024

overrides_exporter:
  replicas: 1
  resources:
    limits:
      memory: 128Mi
    requests:
      cpu: 100m
      memory: 128Mi

querier:
  replicas: 40
  resources:
    limits:
      cpu: 8
      memory: 24Gi
    requests:
      cpu: 4
      memory: 12Gi

query_frontend:
  replicas: 6
  resources:
    limits:
      memory: 4Gi
    requests:
      cpu: 2
      memory: 2Gi

query_scheduler:
  resources:
    limits:
      memory: 4Gi
    requests:
      cpu: 2
      memory: 2Gi

ruler:
  replicas: 2
  resources:
    limits:
      memory: 5.7Gi
    requests:
      cpu: 1
      memory: 4Gi

store_gateway:
  podManagementPolicy: Parallel
  persistentVolume:
    size: 1000Gi
  replicas: 36
  resources:
    limits:
      memory: 36Gi
    requests:
      cpu: 6
      memory: 12Gi


gateway:
  enabledNonEnterprise: true
  replicas: 120
  autoscaling:
    enabled: false
  resources:
    limits:
      memory: 256Mi
    requests:
      cpu: 500m
      memory: 64Mi
  service:
    type: LoadBalancer
    annotations:
      networking.gke.io/load-balancer-type: "Internal"
  nginx:
    basicAuth:
      enabled: true

  ingress:
    enabled: true
    ingressClassName: nginx

nginx:
  enabled: false

[charleskorn]

Could you please share the exact error message you're seeing?

I also note that you've changed the value of blocks_storage.bucket_store.bucket_index.max_stale_period to 6 hours, is there a reason for this?

[mustafayildirim]

execution: attempted to read series at index 2048 from ingester chunks stream, but the stream has failed: rpc error: code = Unavailable desc = closing transport due to: connection error: desc = "error reading from server: read tcp 10.244.88.9:53246->10.244.23.11:9095: read: connection reset by peer", received prior goaway: code: NO_ERROR, debug data: "max_age"

Latest change is compactor config from GKE standard-rwo persistent volume class to premium-rwo.
When we query some metrics for 12 hours or 2 days, it gives this error.

[charleskorn]

Are you using Istio by any chance?

And what does 10.244.23.11 refer to? I assume it's an ingester. If so, are there any errors in its logs?

[mustafayildirim]

It's from querier (10.244.88.9) to ingester (10.244.23.11)

This is an isolated cluster; we don't use Istio in it but send Istio metrics here.