Mimir configuration for AZ cluster

[frootima]

Hello Team,
I am new to Mimir and we are working on setting up a Mimir cluster for Az1, Az2, Az3. In each AZ we have the same Prometheus and Mimir.
In the prometehus.yml file remote_write used the cluster endpoint. Member List Port 7946 is open. grpc port no yet opened. We are getting below error:

Failed to send batch, retrying" err="server returned HTTP status 500 Internal Server Error: failed pushing to ingester: rpc error: code = Canceled desc = grpc: the client connection is closing"

Do we need to open a port for 7946 (member list communication) and gRPC port - 9095?

Below is the configuration we are using:

multitenancy_enabled: false
target: all,alertmanager,overrides-exporter

common:
  storage:
    backend: s3
    s3:
      endpoint: s3.XXXXXX
      bucket_name: XXXXX
      access_key_id: XXXXX
      secret_access_key: XXXXX
      insecure: false

blocks_storage:
  backend: s3
  storage_prefix: blocks
  tsdb:
    retention_period: 3h
    dir: /tmp/ingester

memberlist:
        join_members: [AZ1_NODE,AZ2_NODE,AZ3_NODE]


compactor:
  data_dir: /tmp/mimir/compactor
  compaction_interval: 15m
  cleanup_interval: 15m
  sharding_ring:
    instance_addr: AZ3_NODE
    kvstore:
      store: memberlist



distributor:
  ring:
    instance_addr: AZ3_NODE
    kvstore:
      store: memberlist

ingester:
  ring:
    instance_addr: AZ3_NODE
    kvstore:
      store: memberlist
    replication_factor: 1

alertmanager:
  data_dir: /tmp/alertmanager
  external_url: http://AZ3_NODE/alertmanager

ruler_storage:
  backend: s3
  storage_prefix: ruler

ruler:
  rule_path: /tmp/ruler
  alertmanager_url: http://AZ3_NODE/alertmanager
  ring:
    heartbeat_period: 15s
    heartbeat_timeout: 1m

server:
  http_listen_address: "AZ3_NODE"
  http_listen_port: 9009
  log_level: info
  grpc_server_max_recv_msg_size: 104857600
  grpc_server_max_send_msg_size: 104857600
  grpc_server_max_concurrent_streams: 1000

store_gateway:
  sharding_ring:
    kvstore:
      store: memberlist
    replication_factor: 1

limits:
  max_fetched_chunks_per_query: 50000000000
  max_label_names_per_series: 200
  max_global_series_per_user: 500000000000
  ingestion_rate: 1500000000
  ingestion_burst_size: 4000000000

[dimitarvdimitrov]

yes, mimir needs the HTTP, gRPC, and memberlist ports open on all processes. By default those are 8080, 9095, and 7964

[frootima]

Any response team.

[charleskorn]

@frootima are you seeing any issues running with the configuration you posted?

One thing I can see is that you've used AZ3_NODE as the instance_addr or http_listen_address in a bunch of places - unless this is changed appropriately for each node, this will cause issues.

[frootima]

Thank you @charleskorn .

AZ3_NODE given as instance_addr. What i observed that in individual mimir node (get method) I can see the cortex-related metric data. but Mimir load balancer endpoint (http://load-balancer:9009/prometheus), I can see only Prometheus metrics data. How can I get the cortex-related data as well?

[charleskorn]

Are you referring to the metrics for Mimir itself that start with cortex_...? If so, you'll need to configure something (eg. the Grafana Agent or Prometheus running as a remote-write agent) to scrape Mimir's metrics and write them back to Mimir.

[frootima]

yes sir .. i missed that. thank you very much.