Missing data around ingesters memberlist but perfect from store

[catinsnowright]

Hi Team,
I am playing with Mimir to understand if i can use it in my cluster(deploy components - only puppet) - current build pretty much similar to Play-with-mimir article. I am using only one tenant.

5 instances with 5 Grafana Agent Flow mode each, writing node_exporter data to 3 Monolithic mode Mimirs which are behind Nginx load balancer, s3 storage - Minio.

My problem is that after adding mimir datasource i see gaps of missing data for last 12 hours(which i am sure - tsdb default retention time period), but after - data perfectly consistent without any gaps.

I have removed all retention options and i still got 2 hours period with gaps(because of block ranges period), but rest - complete perfect.

Also when i am refreshing panel every time it shows different data.
When i trying to debug - I have added each mimir as a separate source and for the same period of time each mimir has different data. Gaps from different sources covering by different ingesters.

I might not understand if it's by design and the only way to see ingesters data - to use only one igester or with consul.

How can i make grafana shows me panels without missing data when i see the panel in my grafana dashboards?

I have been reading and experimenting with all configuration from issues here in git - but no luck.

Could someone tell me the right direction?

Thank you in advance.

---

At the beginning i have pretty much simple config to make sure most of configuration are default, but after i comes with one suggestion -

#4696 (reply in thread)

So currently i have config:

alertmanager:
    sharding_ring:
        instance_addr: 0.0.0.0
alertmanager_storage:
    backend: s3
    s3:
        access_key_id: mimir_aki
        bucket_name: cluster1-alertmanager
        endpoint: miniostrg.dev-env.default.com
        insecure: true
        secret_access_key: '********'
blocks_storage:
    backend: s3
    bucket_store:
        ignore_blocks_within: 0s
        ignore_deletion_mark_delay: 30m0s
        metadata_cache:
            bucket_index_content_ttl: 3m0s
        sync_dir: /apps/mimir/tsdb-sync
        sync_interval: 5m0s
    s3:
        access_key_id: mimir_aki
        bucket_name: cluster1-tsdb
        endpoint: miniostrg.dev-env.default.com
        insecure: true
        secret_access_key: '********'
    tsdb:
        dir: /apps/mimir/tsdb
compactor:
    cleanup_interval: 10m0s
    compaction_interval: 15m0s
    data_dir: /apps/mimir/compactor
    first_level_compaction_wait_period: 20m0s
    max_closing_blocks_concurrency: 2
    max_opening_blocks_concurrency: 4
    sharding_ring:
        instance_addr: 0.0.0.0
        wait_stability_min_duration: 1m0s
    symbols_flushers_concurrency: 4
distributor:
    ring:
        instance_addr: 0.0.0.0
frontend:
    address: 0.0.0.0
    align_queries_with_step: true
    cache_results: true
    grpc_client_config:
        grpc_compression: snappy
    log_queries_longer_than: 1m0s
    max_outstanding_per_tenant: 4096
    results_cache:
        backend: memcached
        compression: snappy
        memcached:
            addresses: test-dev-cache.default.com:11211
            timeout: 1s
    scheduler_worker_concurrency: 2048
frontend_worker:
    grpc_client_config:
        max_send_msg_size: 419430400
ingester:
    ring:
        instance_addr: 0.0.0.0
        min_ready_duration: 0s
        num_tokens: 512
        tokens_file_path: /apps/mimir/tokens
ingester_client:
    grpc_client_config:
        grpc_compression: snappy
limits:
    compactor_blocks_retention_period: 1y
    compactor_split_groups: 8
    ingestion_burst_size: 350000
    ingestion_rate: 60000
    max_global_series_per_user: 0
    max_label_names_per_series: 60
    native_histograms_ingestion_enabled: true
    ruler_max_rule_groups_per_tenant: 0
    ruler_max_rules_per_rule_group: 0
memberlist:
    bind_addr:
        - 55.55.55.55(here is real machine ip address)
    cluster_label: cluster1_mimir
    join_members:
        - test-cluster1-mm-01.default.com:7946
        - test-cluster1-mm-02.default.com:7946
        - test-cluster1-mm-03.default.com:7946
    max_join_retries: 120
    rejoin_interval: 10s
multitenancy_enabled: false
querier:
    max_concurrent: 2048
    query_ingesters_within: 6h5m0s
    query_store_after: 0s
ruler:
    poll_interval: 2s
    ring:
        heartbeat_period: 2s
        heartbeat_timeout: 10s
        instance_addr: 0.0.0.0
ruler_storage:
    backend: s3
    s3:
        access_key_id: mimir_aki
        bucket_name: cluster1-rules
        endpoint: miniostrg.dev-env.default.com
        insecure: true
        secret_access_key: '********'
runtime_config:
    file: /etc/mimir/runtime_config.yml
server:
    grpc_server_max_concurrent_streams: 1000
    grpc_server_max_connection_age: 2m0s
    grpc_server_max_connection_age_grace: 5m0s
    grpc_server_max_connection_idle: 1m0s
    http_listen_address: 0.0.0.0
    http_listen_port: 9009
    http_server_read_timeout: 3m0s
    http_server_write_timeout: 3m0s
store_gateway:
    sharding_ring:
        instance_addr: 0.0.0.0
        tokens_file_path: /apps/mimir/tokens
target: all,alertmanager

[narqo]

[..] current build pretty much similar to Play-with-mimir article
...
[..] I have added each mimir as a separate source and for the same period of time each mimir has different data. Gaps from different sources covering by different ingesters.

To make sure I understand the setup, do you run ingesters configured with the default replication factor (ingester.ring.replication_factor=3)?

Could you share the details of the configuration you run Mimir with?

[catinsnowright]

[..] current build pretty much similar to Play-with-mimir article
...
[..] I have added each mimir as a separate source and for the same period of time each mimir has different data. Gaps from different sources covering by different ingesters.

To make sure I understand the setup, do you run ingesters configured with the default replication factor (ingester.ring.replication_factor=3)?

Could you share the details of the configuration you run Mimir with?

yes, replication factor is default = 3.
just have updated initial message to make it more organized.

[dimitarvdimitrov]

3 Monolithic mode Mimirs

what you say this do you mean that you have 3 replicas in the same mimir cluster and they share the same memberlist cluster and memberlist cluster label and one write goes to only one writer OR they are 3 completely disjoint clusters that each receives writes?

From the config cluster_label: cluster1_mimir it makes me think you have 3 mimir clusters each with 1 replica. In which case if you want to replicate data to the 3 clusters you also need to have to agents write it 3 times (once to each cluster).

Alternatively you can just use mostly the exact same config (at least same memberlist cluster label) for all 3 replicas and they will be 1 mimir cluster which internally replicas incoming data. Mimir was design with this use case in mind.

[catinsnowright]

Hello, @dimitarvdimitrov

What i am saying i used horizontally scaled suggested schema from documentation:

i have exact the same configuration on all mimir instances(except IP and mimir install directories), this mean
Mimir 1t node - cluster_label: cluster1_mimir
Mimir 2nd node - cluster_label: cluster1_mimir
Mimir 3d node - cluster_label: cluster1_mimir

i used this option - cluster_label: cluster1_mimir because i thought it will fix the issue.

So currently my agents writing all metrics to endpoint like it was suggested here -
https://grafana.com/docs/mimir/latest/references/architecture/components/distributor/#load-balancing-across-distributors

I am using pretty much the same nginx config(only with my ports):

http {
    upstream backend {
        server mimir-1:8080 max_fails=1 fail_timeout=1s;
        server mimir-2:8080 max_fails=1 fail_timeout=1s;
        server mimir-3:8080 max_fails=1 fail_timeout=1s;
    }

    server {
        listen 9009;
        access_log /dev/null;
        client_body_buffer_size  124m;
        client_max_body_size      124m;
        location / {
            proxy_pass http://backend;
        }
    }

I have got 1 agent per host, with remote write destination pointing to nginx's load balancer
Just in case *.river conf:

prometheus.remote_write "mimir" {
  endpoint {
    url = "http://mimir-nginx:9009/api/v1/push"
    send_native_histograms = true
    send_exemplars = true
  }

[catinsnowright]

@dimitarvdimitrov @narqo

Adding screenshots of my data from grafana.

Version:
mimir - latest
Grafana agent flow - latest

[dimitarvdimitrov]

on a second look at the config I think it might have to do with instance_addr: 0.0.0.0 fields in some of the config blocks. Have you tried removing this line from the config, so that the instances can each detect their own IP addresses? The addresses should be different for the 3 replicas.

[catinsnowright]

i set this configuration because i think hosts doesn't have auto discovery or so. basically if i remove it - i will see error that <component_name> cannot assign IP address.

P.S. on mimir memberlist endpoint i see all 3 hosts as a ring.

[narqo]

I'm trying to reproduce the problem locally, running the setup from Play with Mimir (three instances of mimir, that form a ring; and a push agent, which sends metrics to the LB in front of the cluster). So far it works as expected.

1. Could you double-check the ring status of the ingesters (the /ingester/ring endpoint). With RF=3, each instance expected to own 1/3 of the cluster's ring (refer to the "Ownership" column).
2. Also, check each ingester's TSDB (the /ingester/tenants endpoint). Is there maybe anything that indicates the replication is off?

[narqo]

it might have to do with instance_addr: 0.0.0.0 fields in some of the config blocks

Actually, I also suspect this is the root cause. My understanding is that ring.instance_addr is the address the component advertises to other participants of its ring (i.e. an instance can be active in the memberlist, but its component's advertised address can be unreachable to the ring). To investigate this further, try setting the real IP address, on which the instance is accessible within your cluster's network.

Also, could you share the actual error, that shows how the auto-discovery of the ring's instance-addr fails. There might be a bug to investigate separately.

[catinsnowright]

Thank you so much @narqo @dimitarvdimitrov that was an issue!