Scrape commit failed" err="write to WAL: log samples: write data/wal/XXXXXXXX: no space left on device

[edtshuma]

I am trying to spin-up the Mimir docker sample available here but I am facing a number of issues starting with the Prometheus container:

ts=2024-05-10T07:00:42.970Z caller=manager.go:1009 level=info component="rule manager" msg="Starting rule manager..."
ts=2024-05-10T07:00:48.979Z caller=compact.go:523 level=info component=tsdb msg="write block" mint=1714629600000 maxt=1714636800000 ulid=01HXGMVW5FMTRTS49PFRB59NQ6 duration=36.562625ms
ts=2024-05-10T07:00:48.982Z caller=head.go:1298 level=info component=tsdb msg="Head GC completed" caller=truncateMemory duration=1.66275ms
ts=2024-05-10T07:00:48.982Z caller=checkpoint.go:100 level=info component=tsdb msg="Creating checkpoint" from_segment=33 to_segment=34 mint=1714636800000
ts=2024-05-10T07:00:49.019Z caller=head.go:1266 level=info component=tsdb msg="WAL checkpoint complete" first=33 last=34 duration=37.182875ms
ts=2024-05-10T07:01:37.410Z caller=scrape.go:1344 level=error component="scrape manager" scrape_pool=prometheus target=http://localhost:9090/metrics msg="Scrape commit failed" err="write to WAL: log samples: write data/wal/00000037: no space left on device"
ts=2024-05-10T07:01:38.938Z caller=scrape.go:1344 level=error component="scrape manager" scrape_pool=app-a target=http://app-a:8000/metrics msg="Scrape commit failed" err="write to WAL: log samples: write data/wal/00000037: no space left on device"
ts=2024-05-10T07:01:42.410Z caller=scrape.go:1344 level=error component="scrape manager" scrape_pool=prometheus target=http://localhost:9090/metrics msg="Scrape commit failed" err="write to WAL: log samples: write data/wal/00000037: no space left on device"
ts=2024-05-10T07:01:43.934Z caller=scrape.go:1344 level=error component="scrape manager" scrape_pool=app-a target=http://app-a:8000/metrics msg="Scrape commit failed" err="write to WAL: log samples: write data/wal/00000037: no space left on device"
ts=2024-05-10T09:00:50.310Z caller=db.go:984 level=error component=tsdb msg="compaction failed" err="compact head: persist head block: populate block: write chunks: preallocate: no space left on device"
ts=2024-05-10T09:00:53.786Z caller=db.go:984 level=error component=tsdb msg="compaction failed" err="compact head: persist head block: populate block: write chunks: preallocate: no space left on device"```

Mimir container logs:

```ts=2024-05-10T08:56:53.197459588Z caller=featurecontrol.go:97 level=warn msg="Classic mode enabled"
ts=2024-05-10T10:51:41.838524708Z caller=basic_lifecycler_delegates.go:147 level=warn msg="auto-forgetting instance from the ring because it is unhealthy for a long time" instance=mimir-3 last_heartbeat="2024-05-10 10:48:11 +0000 UTC" forget_period=20s
ts=2024-05-10T10:51:41.843039625Z caller=basic_lifecycler_delegates.go:147 level=warn msg="auto-forgetting instance from the ring because it is unhealthy for a long time" instance=mimir-1 last_heartbeat="2024-05-10 10:48:11 +0000 UTC" forget_period=20s
ts=2024-05-10T10:51:41.843081292Z caller=basic_lifecycler_delegates.go:147 level=warn msg="auto-forgetting instance from the ring because it is unhealthy for a long time" instance=mimir-2 last_heartbeat="2024-05-10 10:48:11 +0000 UTC" forget_period=20s
ts=2024-05-10T10:51:43.892482626Z caller=basic_lifecycler.go:440 level=warn msg="instance is missing in the ring (e.g. the ring backend storage has been reset), registering the instance with an updated registration timestamp" ring=ruler
ts=2024-05-10T10:51:47.080631377Z caller=gateway.go:311 level=warn msg="failed to synchronize TSDB blocks" reason=ring-change err="unable to check tenants owned by this store-gateway instance: store-gateway is unhealthy in the ring"```


I am running on a Mac system and I have checked the space:

` ``df -hi 
Filesystem                                          Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/disk3s3s1                                     460Gi   9.5Gi   295Gi     4%    404k  3.1G    0%   /
devfs                                              209Ki   209Ki     0Bi   100%     722     0  100%   /dev
/dev/disk3s6                                       460Gi    20Ki   295Gi     1%       0  3.1G    0%   /System/Volumes/VM
/dev/disk3s4                                       460Gi   5.7Gi   295Gi     2%    1.1k  3.1G    0%   /System/Volumes/Preboot
/dev/disk3s2                                       460Gi    41Mi   295Gi     1%      50  3.1G    0%   /System/Volumes/Update
/dev/disk1s2                                       500Mi   6.0Mi   481Mi     2%       1  4.9M    0%   /System/Volumes/xarts
/dev/disk1s1                                       500Mi   6.1Mi   481Mi     2%      33  4.9M    0%   /System/Volumes/iSCPreboot
/dev/disk1s3                                       500Mi   2.4Mi   481Mi     1%      49  4.9M    0%   /System/Volumes/Hardware
/dev/disk3s1                                       460Gi   149Gi   295Gi    34%    367k  3.1G    0%   /System/Volumes/Data
map auto_home                                        0Bi     0Bi     0Bi   100%       0     0     -   /System/Volumes/Data/home```

This is how my docker-compose file looks:

```version: "3.4"
services:
  minio:
    image: minio/minio
    entrypoint: [""]
    command: ["sh", "-c", "mkdir -p /data/mimir && minio server --quiet /data"]
    environment:
      - MINIO_ROOT_USER=mimir
      - MINIO_ROOT_PASSWORD=supersecret
    volumes:
      - minio-data:/data

  grafana:
    image: grafana/grafana:latest
    pull_policy: always
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_USERS_DEFAULT_THEME=light
      - GF_LOG_MODE=console
      - GF_LOG_LEVEL=critical
    volumes:
      - ./config/grafana-provisioning-dashboards.yaml:/etc/grafana/provisioning/dashboards/provisioning-dashboards.yaml:ro
      - ./config/grafana-provisioning-datasources.yaml:/etc/grafana/provisioning/datasources/provisioning-datasources.yaml:ro
      # Explicitly list the dashboards we want to show in the demo. We intentionally exclude dashboards that require
      # Kubernetes metrics (eg. resources or networking) and other services not available in the demo (eg. Grafana Loki).
      - ../../../../../operations/mimir-mixin-compiled/dashboards/mimir-alertmanager.json:/var/lib/grafana/dashboards/mimir-alertmanager.json:ro
      - ../../../../../operations/mimir-mixin-compiled/dashboards/mimir-compactor.json:/var/lib/grafana/dashboards/mimir-compactor.json:ro
      - ../../../../../operations/mimir-mixin-compiled/dashboards/mimir-object-store.json:/var/lib/grafana/dashboards/mimir-object-store.json:ro
      - ../../../../../operations/mimir-mixin-compiled/dashboards/mimir-overrides.json:/var/lib/grafana/dashboards/mimir-overrides.json:ro
      - ../../../../../operations/mimir-mixin-compiled/dashboards/mimir-queries.json:/var/lib/grafana/dashboards/mimir-queries.json:ro
      - ../../../../../operations/mimir-mixin-compiled/dashboards/mimir-reads.json:/var/lib/grafana/dashboards/mimir-reads.json:ro
      - ../../../../../operations/mimir-mixin-compiled/dashboards/mimir-ruler.json:/var/lib/grafana/dashboards/mimir-ruler.json:ro
      - ../../../../../operations/mimir-mixin-compiled/dashboards/mimir-tenants.json:/var/lib/grafana/dashboards/mimir-tenants.json:ro
      - ../../../../../operations/mimir-mixin-compiled/dashboards/mimir-top-tenants.json:/var/lib/grafana/dashboards/mimir-top-tenants.json:ro
      - ../../../../../operations/mimir-mixin-compiled/dashboards/mimir-writes.json:/var/lib/grafana/dashboards/mimir-writes.json:ro
    ports:
      - 9000:3000

  prometheus:
    image: prom/prometheus:latest
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --log.level=error
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention=60d
      - --storage.tsdb.wal-compression
      - --web.console.libraries=/usr/share/prometheus/console_libraries
      - --web.console.templates=/usr/share/prometheus/consoles
    volumes:
      - ./config/prometheus.yaml:/etc/prometheus/prometheus.yml
      - ../../../../../operations/mimir-mixin-compiled/rules.yaml:/etc/prometheus/rules.yaml
    depends_on:
      - "mimir-1"
      - "mimir-2"
      - "mimir-3"

  load-balancer:
    image: nginx:latest
    volumes:
      - ./config/nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - "mimir-1"
      - "mimir-2"
      - "mimir-3"
    ports:
      - 9009:9009

  mimir-1:
    image: grafana/mimir:latest
    command: ["-config.file=/etc/mimir.yaml"]
    hostname: mimir-1
    depends_on:
      - minio
    volumes:
      - ./config/mimir.yaml:/etc/mimir.yaml
      - ./config/alertmanager-fallback-config.yaml:/etc/alertmanager-fallback-config.yaml
      - mimir-1-data:/data

  mimir-2:
    image: grafana/mimir:latest
    command: ["-config.file=/etc/mimir.yaml"]
    hostname: mimir-2
    depends_on:
      - minio
    volumes:
      - ./config/mimir.yaml:/etc/mimir.yaml
      - ./config/alertmanager-fallback-config.yaml:/etc/alertmanager-fallback-config.yaml
      - mimir-2-data:/data

  mimir-3:
    image: grafana/mimir:latest
    command: ["-config.file=/etc/mimir.yaml"]
    hostname: mimir-3
    depends_on:
      - minio
    volumes:
      - ./config/mimir.yaml:/etc/mimir.yaml
      - ./config/alertmanager-fallback-config.yaml:/etc/alertmanager-fallback-config.yaml
      - mimir-3-data:/data

volumes:
  mimir-1-data:
  mimir-2-data:
  mimir-3-data:
  minio-data:

I have maintained all the default settings and followed the setup guide available here.

I tried adding the following setting to the Prometheus server but I doesnt solve anything:

- --storage.tsdb.retention=60d - --storage.tsdb.wal-compression

What am I missing ?

[aknuds1]

This should be a discussion, not a bug report.