Sometimes Mimir Ingester is be OOM Killed

[kingbj940429]

I use mimir-distributed helm chart v5.3.0

Sometimes mimir-ingester is be OOM Killed.

This is my values.yaml

ingester:
  persistentVolume:
    enabled: true
    size: 1Ei
    storageClass: efs-sc

  replicas: 3

  resources:
    limits:
      memory: 9Gi
    requests:
      cpu: 2.5
      memory: 9Gi
      
      ...
      
mimir:
  # For more information see https://grafana.com/docs/mimir/latest/references/configuration-parameters/
  structuredConfig:
    ingester:
      ring:
        replication_factor: 3

    alertmanager:
      sharding_ring:
        replication_factor: 1

    store_gateway:
      sharding_ring:
        replication_factor: 2

    overrides_exporter:
      enabled_metrics: ingestion_rate,ingestion_burst_size,max_global_series_per_user,max_global_series_per_metric,max_global_exemplars_per_user,max_fetched_chunks_per_query,max_fetched_series_per_query,max_fetched_chunk_bytes_per_query,ruler_max_rules_per_rule_group,ruler_max_rule_groups_per_tenant,max_global_metadata_per_user,max_global_metadata_per_metric,request_rate,request_burst_size,alertmanager_notification_rate_limit,alertmanager_max_dispatcher_aggregation_groups,alertmanager_max_alerts_count,alertmanager_max_alerts_size_bytes

    #-- For more information see https://grafana.com/docs/mimir/latest/configure/configuration-parameters/#limits
    limits:
      max_global_series_per_user: 5000000
      ingestion_rate: 300000
      max_label_names_per_series: 100
      compactor_blocks_retention_period: 30d

And EKS is small. I have about 200 pods with 20 nodes

This is my mimir-ingester metrics

Etc Informations

Distributor
estimated rate = (sum(prometheus_tsdb_head_series) * (60 / 30)) / 60 = 85,222

Ingester
total number of in-memory series = sum(prometheus_tsdb_head_series) * 3 = 7,670,400

At some point, it suddenly becomes OOM Killed, but I have no idea why.