Problem with DoBatch: InstancesCount

[kssrao87]

Configured mimir on kubernetes with backend to s3 by changing values.yaml file under https://github.com/grafana/mimir/blob/main/operations/helm/charts/mimir-distributed/values.yaml, trying to push data from prometheus to mimir.
Configuration in prometheus:

remote_write:
      - url: "http://mimir-distributor:8080/api/v1/push"
        send_exemplars: true
        queue_config:
          max_shards: 1
          max_samples_per_send: 5000
          capacity: 10000
          batch_send_deadline: 60s
        metadata_config:
          send: false

Received the below error from prometheus pod:

{"caller":"dedupe.go:112","component":"remote","err":"server returned HTTP status 500 Internal Server Error: DoBatch: InstancesCount <= 0","level":"warn","msg":"Failed to send batch, retrying","remote_name":"2715a6","ts":"2023-04-03T14:18:10.382Z","url":"http://mimir-distributor:8080/api/v1/push"}

port-forwarded mimir-distributor pod:

Distributor deployment values from values.yaml:

distributor:
  replicas: 1

  service:
    annotations: {}
    labels: {}

  resources:
    requests:
      cpu: 100m
      memory: 512Mi

  # Additional distributor container arguments, e.g. log level (debug, info, warn, error)
  extraArgs:
    log.level: debug

  # Pod Labels
  podLabels:
    egress-istiod: "true"
    egress-k8s-dns: "true"
    istio-egressgateway-access: "true"

  # Pod Annotations
  podAnnotations:
    mhub.istio.io/version: 1.15.5
    sidecar.istio.io/inject: "true"
    sidecar.istio.io/proxyCPU: 40m
    sidecar.istio.io/proxyMemory: 80Mi

  # Pod Disruption Budget
  podDisruptionBudget: {}

  # -- The name of the PriorityClass for distributor pods
  priorityClassName: null

  nodeSelector: {}
  affinity: {}

  # -- topologySpreadConstraints allows to customize part of the default topologySpreadConstraints. labelSelectors
  # is generated by the chart using the same selectors as for services.
  topologySpreadConstraints:
    maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway

  annotations: {}
  persistence:
    subPath:

  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45

  securityContext:
    fsGroup: 10001
    runAsGroup: 10001
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile:
      type: RuntimeDefault

  # -- The SecurityContext for distributor containers
  containerSecurityContext:
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop: [ALL]

  strategy:
    type: RollingUpdate

  terminationGracePeriodSeconds: 60

  tolerations: []
  initContainers: []
  extraContainers: []
  extraVolumes: []
  extraVolumeMounts: []
  env: []
  extraEnvFrom: []

Ingester sts values from values.yaml:

ingester:
  # -- Total number of replicas for the ingester across all availability zones
  # If ingester.zoneAwareReplication.enabled=false, this number is taken as is.
  # Otherwise each zone starts `ceil(replicas / number_of_zones)` number of pods.
  #   E.g. if 'replicas' is set to 4 and there are 3 zones, then 4/3=1.33 and after rounding up it means 2 pods per zone are started.
  replicas: 3

  statefulSet:
    enabled: true

  service:
    annotations: {}
    labels: {}

  # -- Optionally set the scheduler for pods of the ingester
  schedulerName: ""

  resources:
    requests:
      cpu: 100m
      memory: 512Mi

  # Additional ingester container arguments, e.g. log level (debug, info, warn, error)
  extraArgs:
    log.level: debug

  # Pod Labels
  podLabels:
    egress-istiod: "true"
    egress-k8s-dns: "true"
    istio-egressgateway-access: "true"

  # Pod Annotations
  podAnnotations:
    mhub.istio.io/version: 1.15.5
    sidecar.istio.io/inject: "true"
    sidecar.istio.io/proxyCPU: 40m
    sidecar.istio.io/proxyMemory: 80Mi
    traffic.sidecar.istio.io/includeInboundPorts: '*'
    traffic.sidecar.istio.io/includeOutboundIPRanges: '*'
  # -- The name of the PriorityClass for ingester pods
  priorityClassName: null

  # -- Pod Disruption Budget for ingester, this will be applied across availability zones to prevent losing redundancy
  podDisruptionBudget:
    maxUnavailable: 1

  podManagementPolicy: Parallel

  # -- NodeSelector to pin ingester pods to certain set of nodes. This is ignored when ingester.zoneAwareReplication.enabled=true.
  nodeSelector: {}
  # -- Pod affinity settings for the ingester. This is ignored when ingester.zoneAwareReplication.enabled=true.
  affinity: {}

  # -- topologySpreadConstraints allows to customize part of the default topologySpreadConstraints. labelSelectors
  # is generated by the chart using the same selectors as for services.
  # It is recommended to replace this with requiredDuringSchedulingIgnoredDuringExecution podAntiAffinity rules when
  # deploying to production.
  topologySpreadConstraints:
    maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway

  annotations: {}

  persistentVolume:
    # If true and ingester.statefulSet.enabled is true,
    # Ingester will create/use a Persistent Volume Claim
    # If false, use emptyDir
    #
    enabled: false

    # Ingester data Persistent Volume Claim annotations
    #
    annotations: {}

    # Ingester data Persistent Volume access modes
    # Must match those of existing PV or dynamic provisioner
    # Ref: http://kubernetes.io/docs/user-guide/persistent-volumes/
    accessModes:
      - ReadWriteOnce

    # Ingester data Persistent Volume size
    size: 2Gi

    # Subdirectory of Ingester data Persistent Volume to mount
    # Useful if the volume's root directory is not empty
    subPath: ''


    # Ingester data Persistent Volume Storage Class
    # If defined, storageClassName: <storageClass>
    # If set to "-", storageClassName: "", which disables dynamic provisioning
    # If undefined (the default) or set to null, no storageClassName spec is
    #   set, choosing the default provisioner.  (gp2 on AWS, standard on
    #   GKE, AWS & OpenStack)
    #
    # storageClass: "-"

  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 60

  securityContext:
    fsGroup: 10001
    runAsGroup: 10001
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile:
      type: RuntimeDefault

  # -- The SecurityContext for ingester containers
  containerSecurityContext:
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop: [ALL]

  # -- updateStrategy of the ingester statefulset. This is ignored when ingester.zoneAwareReplication.enabled=true.
  statefulStrategy:
    type: RollingUpdate

  terminationGracePeriodSeconds: 240

  tolerations: []
  initContainers: []
  extraContainers: []
  extraVolumes: []
  extraVolumeMounts: []
  env: []
  extraEnvFrom: []

  # -- Options to configure zone-aware replication for ingester
  # Example configuration with full geographical redundancy:
  # rollout_operator:
  #   enabled: true
  # ingester:
  #   zoneAwareReplication:
  #     enabled: true
  #     topologyKey: 'kubernetes.io/hostname'  # This generates default anti-affinity rules
  #     zones:  # Zone list has to be fully redefined for modification. Update with you actual zones or skip to use logical zones only.
  #     - name: zone-a
  #       nodeSelector:
  #         topology.kubernetes.io/zone: us-central1-a
  #     - name: zone-a
  #       nodeSelector:
  #         topology.kubernetes.io/zone: us-central1-b
  #     - name: zone-c
  #       nodeSelector:
  #         topology.kubernetes.io/zone: us-central1-c
  #
  zoneAwareReplication:
    # -- Enable zone-aware replication for ingester
    enabled: false
    # -- Maximum number of ingesters that can be unavailable per zone during rollout
    maxUnavailable: 25
    # -- topologyKey to use in pod anti-affinity. If unset, no anti-affinity rules are generated. If set, the generated anti-affinity rule makes sure that pods from different zones do not mix.
    # E.g.: topologyKey: 'kubernetes.io/hostname'
    topologyKey: null
    # -- Auxiliary values for migration, see https://grafana.com/docs/mimir/latest/migration-guide/migrating-from-single-zone-with-helm/
    migration:
      # -- Indicate if migration is ongoing for multi zone ingester
      enabled: false
      # -- Exclude default zone on write path
      excludeDefaultZone: false
      # -- Enable zone-awareness, read path only
      readPath: false
      # -- Total number of replicas to start in availability zones when migration is enabled
      replicas: 0
      # -- Scale default zone ingesters to 0
      scaleDownDefaultZone: false
      # -- Enable zone-awareness, write path only
      writePath: false
    # -- Zone definitions for ingester zones. Note: you have to redefine the whole list to change parts as YAML does not allow to modify parts of a list.
    zones:
    # -- Name of the zone, used in labels and selectors. Must follow Kubernetes naming restrictions: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
    - name: zone-a
      # -- nodeselector to restrict where pods of this zone can be placed. E.g.:
      # nodeSelector:
      #   topology.kubernetes.io/zone: zone-a
      nodeSelector: null
      # -- extraAffinity adds user defined custom affinity rules (merged with generated rules)
      extraAffinity: {}
    # -- Name of the zone, used in labels and selectors. Must follow Kubernetes naming restrictions: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
    - name: zone-b
      # -- nodeselector to restrict where pods of this zone can be placed. E.g.:
      # nodeSelector:
      #   topology.kubernetes.io/zone: zone-b
      nodeSelector: null
      # -- extraAffinity adds user defined custom affinity rules (merged with generated rules)
      extraAffinity: {}
    # -- Name of the zone, used in labels and selectors. Must follow Kubernetes naming restrictions: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
    - name: zone-c
      # -- nodeselector to restrict where pods of this zone can be placed. E.g.:
      # nodeSelector:
      #   topology.kubernetes.io/zone: zone-c
      nodeSelector: null
      # -- extraAffinity adds user defined custom affinity rules (merged with generated rules)
      extraAffinity: {}

distributor logs have some connectivity issues:

ts=2023-04-05T08:14:35.720058451Z caller=log.go:194 level=debug msg="Failed to join 172.30.75.216:7946: dial tcp 172.30.75.216:7946: connect: connection refused"
ts=2023-04-05T08:14:35.720139907Z caller=memberlist_client.go:549 level=debug msg="fast-joining node failed" node=172.30.75.216:7946 err="1 error occurred:\n\t* Failed to join 172.30.75.216:7946: dial tcp 172.30.75.216:7946: connect: connection refused\n\n"

Please help me what I am missing to make distributor to have InstancesCount > 0

[dimitarvdimitrov]

It looks like you're using istio. I don't have experience with istio, but there is this issue about someone mentioning their experience with Mimir and istio and there were incompatibilities with istio at the time #2852

[kssrao87]

Thank you Dimitar Dimitrov.

[kssrao87]

Sorry @dimitarvdimitrov, I thought GRPC logs are enabled.
Logs from distributor:

2023/04/11 11:03:56 INFO: [core] [Channel #3] Channel created
2023/04/11 11:03:56 INFO: [core] [Channel #3] original dial target is: "172.30.116.201:9095"
2023/04/11 11:03:56 INFO: [core] [Channel #3] dial target "172.30.116.201:9095" parse failed: parse "172.30.116.201:9095": first path segment in URL cannot contain colon
2023/04/11 11:03:56 INFO: [core] [Channel #3] fallback to scheme "passthrough"
2023/04/11 11:03:56 INFO: [core] [Channel #3] parsed dial target is: {Scheme:passthrough Authority: Endpoint:172.30.116.201:9095 URL:{Scheme:passthrough Opaque: User: Host: Path:/172.30.116.201:9095 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/04/11 11:03:56 INFO: [core] [Channel #3] Channel authority set to "172.30.116.201:9095"
2023/04/11 11:03:56 INFO: [core] [Channel #3] Resolver state updated: {
  "Addresses": [
    {
      "Addr": "172.30.116.201:9095",
      "ServerName": "",
      "Attributes": null,
      "BalancerAttributes": null,
      "Type": 0,
      "Metadata": null
    }
  ],
  "ServiceConfig": null,
  "Attributes": null
} (resolver returned new addresses)
2023/04/11 11:03:56 INFO: [core] [Channel #3] Channel switches to new LB policy "pick_first"
2023/04/11 11:03:56 INFO: [core] [Channel #3 SubChannel #4] Subchannel created
2023/04/11 11:03:56 INFO: [core] blockingPicker: the picked transport is not ready, loop back to repick
2023/04/11 11:03:56 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to CONNECTING
2023/04/11 11:03:56 INFO: [core] [Channel #3 SubChannel #4] Subchannel picks a new address "172.30.116.201:9095" to connect
2023/04/11 11:03:56 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc0013f6970, {CONNECTING <nil>}
2023/04/11 11:03:56 INFO: [core] [Channel #3] Channel Connectivity change to CONNECTING
2023/04/11 11:03:56 INFO: [core] [Channel #6] Channel created
2023/04/11 11:03:56 INFO: [core] [Channel #6] original dial target is: "172.30.71.217:9095"
2023/04/11 11:03:56 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to READY
2023/04/11 11:03:56 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc0013f6970, {READY <nil>}
2023/04/11 11:03:56 INFO: [core] [Channel #3] Channel Connectivity change to READY
2023/04/11 11:03:56 INFO: [core] [Channel #6] dial target "172.30.71.217:9095" parse failed: parse "172.30.71.217:9095": first path segment in URL cannot contain colon
2023/04/11 11:03:56 INFO: [core] [Channel #6] fallback to scheme "passthrough"
2023/04/11 11:03:56 INFO: [core] [Channel #6] parsed dial target is: {Scheme:passthrough Authority: Endpoint:172.30.71.217:9095 URL:{Scheme:passthrough Opaque: User: Host: Path:/172.30.71.217:9095 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/04/11 11:03:56 INFO: [core] [Channel #6] Channel authority set to "172.30.71.217:9095"
2023/04/11 11:03:56 INFO: [core] [Channel #6] Resolver state updated: {
  "Addresses": [
    {
      "Addr": "172.30.71.217:9095",
      "ServerName": "",
      "Attributes": null,
      "BalancerAttributes": null,
      "Type": 0,
      "Metadata": null
    }
  ],
  "ServiceConfig": null,
  "Attributes": null
} (resolver returned new addresses)
2023/04/11 11:03:56 INFO: [core] [Channel #6] Channel switches to new LB policy "pick_first"
2023/04/11 11:03:56 INFO: [core] [Channel #6 SubChannel #7] Subchannel created
2023/04/11 11:03:56 INFO: [core] [Channel #6 SubChannel #7] Subchannel Connectivity change to CONNECTING
2023/04/11 11:03:56 INFO: [core] [Channel #6 SubChannel #7] Subchannel picks a new address "172.30.71.217:9095" to connect
2023/04/11 11:03:56 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc0013f6f70, {CONNECTING <nil>}
2023/04/11 11:03:56 INFO: [core] [Channel #6] Channel Connectivity change to CONNECTING
2023/04/11 11:03:56 INFO: [core] blockingPicker: the picked transport is not ready, loop back to repick
2023/04/11 11:03:56 INFO: [core] [Channel #8] Channel created
2023/04/11 11:03:56 INFO: [core] [Channel #8] original dial target is: "172.30.59.255:9095"
2023/04/11 11:03:56 INFO: [core] [Channel #8] dial target "172.30.59.255:9095" parse failed: parse "172.30.59.255:9095": first path segment in URL cannot contain colon
2023/04/11 11:03:56 INFO: [core] [Channel #8] fallback to scheme "passthrough"
2023/04/11 11:03:56 INFO: [core] [Channel #8] parsed dial target is: {Scheme:passthrough Authority: Endpoint:172.30.59.255:9095 URL:{Scheme:passthrough Opaque: User: Host: Path:/172.30.59.255:9095 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/04/11 11:03:56 INFO: [core] [Channel #8] Channel authority set to "172.30.59.255:9095"
2023/04/11 11:03:56 INFO: [core] [Channel #8] Resolver state updated: {
  "Addresses": [
    {
      "Addr": "172.30.59.255:9095",
      "ServerName": "",
      "Attributes": null,
      "BalancerAttributes": null,
      "Type": 0,
      "Metadata": null
    }
  ],
  "ServiceConfig": null,
  "Attributes": null
} (resolver returned new addresses)
2023/04/11 11:03:56 INFO: [core] [Channel #8] Channel switches to new LB policy "pick_first"
2023/04/11 11:03:56 INFO: [core] [Channel #8 SubChannel #10] Subchannel created
2023/04/11 11:03:56 INFO: [core] [Channel #8 SubChannel #10] Subchannel Connectivity change to CONNECTING
2023/04/11 11:03:56 INFO: [core] [Channel #8 SubChannel #10] Subchannel picks a new address "172.30.59.255:9095" to connect
2023/04/11 11:03:56 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc0013f7560, {CONNECTING <nil>}
2023/04/11 11:03:56 INFO: [core] [Channel #8] Channel Connectivity change to CONNECTING
2023/04/11 11:03:56 INFO: [core] [Channel #8 SubChannel #10] Subchannel Connectivity change to READY
2023/04/11 11:03:56 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc0013f7560, {READY <nil>}
2023/04/11 11:03:56 INFO: [core] [Channel #8] Channel Connectivity change to READY
2023/04/11 11:03:57 INFO: [core] [Channel #6 SubChannel #7] Subchannel Connectivity change to READY
2023/04/11 11:03:57 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc0013f6f70, {READY <nil>}
2023/04/11 11:03:57 INFO: [core] [Channel #6] Channel Connectivity change to READY
ts=2023-04-11T11:03:57.101211528Z caller=logging.go:86 level=warn traceID=138088a641a6f917 msg="POST /api/v1/push (500) 275.36284ms Response: \"failed pushing to ingester: rpc error: code = Unavailable desc = \\n\" ws: false; Content-Encoding: snappy; Content-Length: 205896; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.43.0; X-B3-Sampled: 0; X-B3-Spanid: b28ebc8b26fb7b38; X-B3-Traceid: 787a2524b589bdf8b28ebc8b26fb7b38; X-Envoy-Attempt-Count: 1; X-Forwarded-Client-Cert: By=spiffe://cluster.local/ns/default/sa/mimir;Hash=4dd648530a8c37385e3b7d9ba94e3733f4c01d10587cda065a22df36dfe45f7c;Subject=\"\";URI=spiffe://cluster.local/ns/default/sa/prometheus; X-Forwarded-Proto: http; X-Prometheus-Remote-Write-Version: 0.1.0; X-Request-Id: b9b83924-7491-4f58-9bac-850028148b6a; "
ts=2023-04-11T11:03:57.378611485Z caller=logging.go:86 level=warn traceID=0802e05775587232 msg="POST /api/v1/push (500) 126.642378ms Response: \"failed pushing to ingester: rpc error: code = Unavailable desc = \\n\" ws: false; Content-Encoding: snappy; Content-Length: 205896; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.43.0; X-B3-Sampled: 0; X-B3-Spanid: ad86d8fdd5e0b1a5; X-B3-Traceid: 5d926e71dcbbf7b2ad86d8fdd5e0b1a5; X-Envoy-Attempt-Count: 1; X-Forwarded-Client-Cert: By=spiffe://cluster.local/ns/default/sa/mimir;Hash=4dd648530a8c37385e3b7d9ba94e3733f4c01d10587cda065a22df36dfe45f7c;Subject=\"\";URI=spiffe://cluster.local/ns/default/sa/prometheus; X-Forwarded-Proto: http; X-Prometheus-Remote-Write-Version: 0.1.0; X-Request-Id: 0cd5112d-8799-4af4-8b99-22b469134201; "
ts=2023-04-11T11:03:57.495813363Z caller=logging.go:86 level=warn traceID=33512c98cdd0dac8 msg="POST /api/v1/push (500) 51.232798ms Response: \"failed pushing to ingester: rpc error: code = Unavailable desc = \\n\" ws: false; Content-Encoding: snappy; Content-Length: 205896; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.43.0; X-B3-Sampled: 0; X-B3-Spanid: 1b174a9ae2019cd6; X-B3-Traceid: 879edc5fff820d721b174a9ae2019cd6; X-Envoy-Attempt-Count: 1; X-Forwarded-Client-Cert: By=spiffe://cluster.local/ns/default/sa/mimir;Hash=4dd648530a8c37385e3b7d9ba94e3733f4c01d10587cda065a22df36dfe45f7c;Subject=\"\";URI=spiffe://cluster.local/ns/default/sa/prometheus; X-Forwarded-Proto: http; X-Prometheus-Remote-Write-Version: 0.1.0; X-Request-Id: 21fccae4-ee60-4b0f-a5f6-7bc620e2c51d; "
ts=2023-04-11T11:03:57.715546699Z caller=logging.go:86 level=warn traceID=6fc688211fe53823 msg="POST /api/v1/push (500) 88.992434ms Response: \"failed pushing to ingester: rpc error: code = Unavailable desc = \\n\" ws: false; Content-Encoding: snappy; Content-Length: 205896; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.43.0; X-B3-Sampled: 0; X-B3-Spanid: 07ea1ba7e8f6869e; X-B3-Traceid: fcd1e8aae87c256107ea1ba7e8f6869e; X-Envoy-Attempt-Count: 1; X-Forwarded-Client-Cert: By=spiffe://cluster.local/ns/default/sa/mimir;Hash=4dd648530a8c37385e3b7d9ba94e3733f4c01d10587cda065a22df36dfe45f7c;Subject=\"\";URI=spiffe://cluster.local/ns/default/sa/prometheus; X-Forwarded-Proto: http; X-Prometheus-Remote-Write-Version: 0.1.0; X-Request-Id: 6b201ae8-25c2-4caf-9ce4-7614bed214b7; "
ts=2023-04-11T11:03:58.023684109Z caller=logging.go:86 level=warn traceID=5969e968ba0bc3a9 msg="POST /api/v1/push (500) 42.154673ms Response: \"failed pushing to ingester: rpc error: code = Unavailable desc = \\n\" ws: false; Content-Encoding: snappy; Content-Length: 205896; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.43.0; X-B3-Sampled: 0; X-B3-Spanid: a859ead894d8dc67; X-B3-Traceid: 75a026ebd6314702a859ead894d8dc67; X-Envoy-Attempt-Count: 1; X-Forwarded-Client-Cert: By=spiffe://cluster.local/ns/default/sa/mimir;Hash=4dd648530a8c37385e3b7d9ba94e3733f4c01d10587cda065a22df36dfe45f7c;Subject=\"\";URI=spiffe://cluster.local/ns/default/sa/prometheus; X-Forwarded-Proto: http; X-Prometheus-Remote-Write-Version: 0.1.0; X-Request-Id: b8e51076-708f-4601-9ce1-41f605d12a29; "
ts=2023-04-11T11:03:58.574220685Z caller=logging.go:86 level=warn traceID=28e4a7d056f9f9d0 msg="POST /api/v1/push (500) 62.585272ms Response: \"failed pushing to ingester: rpc error: code = Unavailable desc = \\n\" ws: false; Content-Encoding: snappy; Content-Length: 205896; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.43.0; X-B3-Sampled: 0; X-B3-Spanid: 708fd4bc0fe2cf09; X-B3-Traceid: 4fb2a1ce0fc8f167708fd4bc0fe2cf09; X-Envoy-Attempt-Count: 1; X-Forwarded-Client-Cert: By=spiffe://cluster.local/ns/default/sa/mimir;Hash=4dd648530a8c37385e3b7d9ba94e3733f4c01d10587cda065a22df36dfe45f7c;Subject=\"\";URI=spiffe://cluster.local/ns/default/sa/prometheus; X-Forwarded-Proto: http; X-Prometheus-Remote-Write-Version: 0.1.0; X-Request-Id: c1ac37cd-18d9-4c7c-8dc4-cb3a1157bd1c; "
ts=2023-04-11T11:03:59.58530558Z caller=logging.go:86 level=warn traceID=18787370ee397eae msg="POST /api/v1/push (500) 46.416291ms Response: \"failed pushing to ingester: rpc error: code = Unavailable desc = \\n\" ws: false; Content-Encoding: snappy; Content-Length: 205896; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.43.0; X-B3-Sampled: 0; X-B3-Spanid: 1cf9793c677682c9; X-B3-Traceid: 750dae0f090be9251cf9793c677682c9; X-Envoy-Attempt-Count: 1; X-Forwarded-Client-Cert: By=spiffe://cluster.local/ns/default/sa/mimir;Hash=4dd648530a8c37385e3b7d9ba94e3733f4c01d10587cda065a22df36dfe45f7c;Subject=\"\";URI=spiffe://cluster.local/ns/default/sa/prometheus; X-Forwarded-Proto: http; X-Prometheus-Remote-Write-Version: 0.1.0; X-Request-Id: 052a1a49-93d5-48de-be39-e00fca73a9df; "
ts=2023-04-11T11:04:01.49456483Z caller=logging.go:76 level=debug traceID=44fd669a01d0d885 msg="GET /ready (200) 144.824µs"
ts=2023-04-11T11:04:01.552876936Z caller=logging.go:86 level=warn traceID=71c80de1b5d07a89 msg="POST /api/v1/push (500) 43.876942ms Response: \"failed pushing to ingester: rpc error: code = Unavailable desc = \\n\" ws: false; Content-Encoding: snappy; Content-Length: 205896; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.43.0; X-B3-Sampled: 0; X-B3-Spanid: d8cf5fd28d4bbb6d; X-B3-Traceid: e9945ac9735ad072d8cf5fd28d4bbb6d; X-Envoy-Attempt-Count: 1; X-Forwarded-Client-Cert: By=spiffe://cluster.local/ns/default/sa/mimir;Hash=4dd648530a8c37385e3b7d9ba94e3733f4c01d10587cda065a22df36dfe45f7c;Subject=\"\";URI=spiffe://cluster.local/ns/default/sa/prometheus; X-Forwarded-Proto: http; X-Prometheus-Remote-Write-Version: 0.1.0; X-Request-Id: 3e9794d7-7904-48a3-b255-4db270e1b07b; "
ts=2023-04-11T11:04:05.448070977Z caller=logging.go:86 level=warn traceID=22cd50bbbfe8e580 msg="POST /api/v1/push (500) 50.891963ms Response: \"failed pushing to ingester: rpc error: code = Unavailable desc = \\n\" ws: false; Content-Encoding: snappy; Content-Length: 205896; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.43.0; X-B3-Sampled: 0; X-B3-Spanid: 81a2bf1b0ef509a6; X-B3-Traceid: 23768c1b443fdd0c81a2bf1b0ef509a6; X-Envoy-Attempt-Count: 1; X-Forwarded-Client-Cert: By=spiffe://cluster.local/ns/default/sa/mimir;Hash=4dd648530a8c37385e3b7d9ba94e3733f4c01d10587cda065a22df36dfe45f7c;Subject=\"\";URI=spiffe://cluster.local/ns/default/sa/prometheus; X-Forwarded-Proto: http; X-Prometheus-Remote-Write-Version: 0.1.0; X-Request-Id: 1767d6c2-25f6-42fb-b841-f7a9fdb12a89; "
ts=2023-04-11T11:04:08.013761833Z caller=log.go:194 level=debug msg="Stream connection from=127.0.0.6:49197"
ts=2023-04-11T11:04:08.224376674Z caller=log.go:194 level=debug msg="Initiating push/pull sync with: mimir-ruler-748bd779ff-6wkc4-a7b7077d 172.30.59.212:7946"
ts=2023-04-11T11:04:08.517655334Z caller=pool.go:193 level=warn msg="removing ingester failing healthcheck" addr=172.30.59.255:9095 reason="rpc error: code = Unavailable desc = "
ts=2023-04-11T11:04:08.517655334Z caller=pool.go:193 level=warn msg="removing ingester failing healthcheck" addr=172.30.71.217:9095 reason="rpc error: code = Unavailable desc = "
2023/04/11 11:04:08 INFO: [core] [Channel #6] Channel Connectivity change to SHUTDOWN
2023/04/11 11:04:08 INFO: [core] [Channel #6 SubChannel #7] Subchannel Connectivity change to SHUTDOWN
2023/04/11 11:04:08 INFO: [core] [Channel #6 SubChannel #7] Subchannel deleted
2023/04/11 11:04:08 INFO: [core] [Channel #6] Channel deleted
ts=2023-04-11T11:04:08.517700681Z caller=pool.go:193 level=warn msg="removing ingester failing healthcheck" addr=172.30.116.201:9095 reason="rpc error: code = Unavailable desc = "
2023/04/11 11:04:08 INFO: [transport] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2023/04/11 11:04:08 INFO: [core] [Channel #8] Channel Connectivity change to SHUTDOWN
2023/04/11 11:04:08 INFO: [core] [Channel #8 SubChannel #10] Subchannel Connectivity change to SHUTDOWN
2023/04/11 11:04:08 INFO: [core] [Channel #8 SubChannel #10] Subchannel deleted
2023/04/11 11:04:08 INFO: [core] [Channel #3] Channel Connectivity change to SHUTDOWN
2023/04/11 11:04:08 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to SHUTDOWN
2023/04/11 11:04:08 INFO: [core] [Channel #3 SubChannel #4] Subchannel deleted
2023/04/11 11:04:08 INFO: [core] [Channel #3] Channel deleted
2023/04/11 11:04:08 INFO: [core] [Channel #8] Channel deleted
2023/04/11 11:04:08 INFO: [transport] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2023/04/11 11:04:08 INFO: [transport] transport: loopyWriter.run returning. connection error: desc = "transport is closing"

Note: When I disabled istio, flow from prometheus to nginx/distributor -> ingester -> cos is working great. But when I enabled istio only I am getting the above grpc error logs, but the flow from prometheus to ingester -> cos is working.

[dimitarvdimitrov]

I don't have many ideas where to go next. Are there any correlated logs on the ingester side? For example, what did the ingester logs of ingester 172.30.59.255 say at around 2023-04-11T11:04Z?

[kssrao87]

I didn't see any error logs from ingester, seems request is not being reached to ingester.

[jabeerkhan85]

hi Siva , am doing the POC on mimir with istio .. as per reading looks like it is not recommended (am still in initial face though ) ..pls let me know your findings that can help me.

[alextricity25]

as per reading looks like it is not recommended (am still in initial face though ) ..pls let me know your findings that can help me.

Where did you see this?