Why is jaeger so obtuse with configuration?

[ekimminau]

Im a hair width from just tossing jaeger into the bin. Im trying to use jaeger in my docker dekstop environment. The configuration is so over the top picky about formatting, syntax and placement that a configuration checker that actually helps you fix issues is REALLY needed. Ive done everything I can possibly think of to disable ipv6 but it keeps coming back and the health checks require more hoops than I can count. Getting otel to feed to prometheus and for jaeger to properly see the traces has been a multi-day excursion through hell. The slightest change breaks one or more flows and I have to struggle through figuring out what broke where. Very frustrating.

Jaeger service from docker-compose.yml:

Jaeger Distributed Tracing (v2) - Now receives traces from OTEL Collector with SPM

  jaeger:
    build:
      context: ./jaeger
      dockerfile: Dockerfile
    image: jaeger:2.9.0
    container_name: tracing
    env_file:
      - .env
    environment:
      - TZ=${TZ}
      - LOG_LEVEL=info
      - QUERY_SERVER_HTTP_PORT=${QUERY_SERVER_HTTP_PORT}
      - QUERY_SERVER_HTTP_HOST_PORT=${QUERY_SERVER_HTTP_HOST_PORT}
      - OTEL_TRACES_SAMPLER=always_off  # Disable Jaeger self-tracing to prevent localhost:4317 attempts
      - OTEL_TRACES_EXPORTER=none  # Disable Jaeger internal self-export to OTLP; collector handles ingestion
      - OTEL_METRICS_EXPORTER=none  # Prevent default SDK metrics exporter from attempting localhost:4317
      - OTEL_LOGS_EXPORTER=none     # Prevent logs exporter attempts
      - OTEL_SDK_DISABLED=true      # Fully disable auto SDK init to stop localhost (::1) OTLP connection attempts
    command: ["--config", "/etc/jaeger/config.yaml"]
    volumes:
      - ./certs:/certs:ro
      - ./jaeger/jaeger-config.yaml:/etc/jaeger/config.yaml:ro
      - ./jaeger/ui-config.json:/etc/jaeger/ui-config.json:ro 
      - jaeger-badger:/badger
      - jaeger-tmp:/tmp
    ports:
      - "16686:16686"  # Jaeger UI (HTTP UI only; internal OTLP secured)
      - "8888:8888"    # Jaeger internal telemetry metrics (Prometheus scrape)
      - "13133:13133"  # Jaeger healthcheckv2 /status endpoint (still loopback internally unless reconfigured)
    restart: unless-stopped
    depends_on:
      traefik:
        condition: service_healthy
      prometheus:
        condition: service_healthy
    networks:
      proxy_network:
        ipv4_address: <IP>
        aliases:
          - jaeger
          - jaeger-tracing
          - jaeger.letsgo.mobi.local
          - tracing.letsgo.mobi.local
      web_network:
        ipv4_address: <IP>
        aliases:
          - jaeger
          - jaeger-tracing
          - jaeger.letsgo.mobi.local
          - tracing.letsgo.mobi.local
    healthcheck:
      # Use GET request (HEAD via --spider fails) and verify expected JSON contains Server available
      test: ["CMD-SHELL", "wget -q -O - http://127.0.0.1:13133/status 2>/dev/null | grep -q 'Server available' || exit 1"]
      interval: 15s
      timeout: 5s
      retries: 5
      start_period: 25s

My Dockerfile:

Custom Jaeger Dockerfile with proper Badger volume permissions

Based on official jaegertracing/jaeger:2.9.0 image

FROM jaegertracing/jaeger:2.9.0

# Temporarily switch to root to set up directories and permissions
USER root

# Create Badger directories with proper ownership and permissions
RUN mkdir -p /badger/keys /badger/values && \
    chown -R 10001:0 /badger && \
    chmod -R 755 /badger

# Switch back to the Jaeger user for security
USER 10001

# Keep the default entrypoint and command behavior
# The --config parameter will be provided by Docker Compose
ENV JAEGER_LISTEN_HOST=0.0.0.0

# Web HTTP
EXPOSE 16686
# Sampling config gRPC
EXPOSE 5779
# Collector gRPC
EXPOSE 14250
# Health Check gRPC
EXPOSE 13132
# Health Check HTTP
EXPOSE 13133
# Prometheus Metrics
EXPOSE 8888

My jaeger-config.yaml:

service:
  # Added healthcheckv2 extension for proper /status endpoint
  extensions: [jaeger_storage, jaeger_query, healthcheckv2]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger_storage_exporter]
  telemetry:
    resource:
      service.name: jaeger
    metrics:
      level: detailed
      readers:
        - pull:
            exporter:
              prometheus:
                host: 127.0.0.1
                port: 8888
    logs:
      level: info

extensions:
  # jaeger_query extension: only storage mapping is valid here (server/ui were invalid keys for v2)
  jaeger_query:
    storage:
      traces: some_storage
      metrics: some_metrics_storage
  jaeger_storage:
    backends:
      some_storage:
        badger:
          directories:
            keys: "/badger/keys"
            values: "/badger/values"
          ephemeral: false
          ttl:
            spans: 48h
    metric_backends:
      some_metrics_storage:
        prometheus:
          endpoint: http://prometheus:9090
          normalize_calls: true
          normalize_duration: true
  # Health endpoint on 13133 (/status). No extra config needed for defaults.
  healthcheckv2: {}

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
        tls:
          cert_file: /certs/letsgo-complete.pem
          key_file: /certs/letsgo-complete-key.pem
      http:
        endpoint: "0.0.0.0:4318"
        tls:
          cert_file: /certs/letsgo-complete.pem
          key_file: /certs/letsgo-complete-key.pem
     
processors:
  batch:

exporters:
  jaeger_storage_exporter:
    trace_storage: some_storage

The error messages I am getting. I believe these are being caused by otel health checks but how it defaults to using ipv6 even though its disabled is beyond me.:
2025/08/13 17:00:03 traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: read tcp 127.0.0.1:47416->127.0.0.1:4317: read: connection reset by peer"

2025-08-13T17:00:09.361Z warn grpc@v1.74.2/clientconn.go:1414 [core] [Channel #1 SubChannel #8]grpc: addrConn.createTransport failed to connect to {Addr: "[::1]:4317", ServerName: "localhost:4317", }. Err: connection error: desc = "error reading server preface: read tcp [::1]:33798->[::1]:4317: read: connection reset by peer" {"resource": {"service.instance.id": "0b8d7623-5c9d-40a1-963f-6c10395e27e9", "service.name": "jaeger", "service.version": "v2.9.0"}, "grpc_log": true}

2025-08-13T17:00:09.667Z warn grpc@v1.74.2/clientconn.go:1414 [core] [Channel #1 SubChannel #6]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:4317", ServerName: "localhost:4317", }. Err: connection error: desc = "error reading server preface: EOF" {"resource": {"service.instance.id": "0b8d7623-5c9d-40a1-963f-6c10395e27e9", "service.name": "jaeger", "service.version": "v2.9.0"}, "grpc_log": true}

[dosubot[bot]]

Hi @ekimminau! I'm Dosu and I’m helping the jaeger team.

Jaeger's configuration is strict and can be frustrating, especially in Docker. There isn't a standalone config checker or linter—errors only show up at runtime, and even small mistakes (like a hyphen instead of an underscore) can break things without clear guidance on how to fix them [source].

The IPv6 connection attempts you see (e.g., [::1]:4317) are usually from Jaeger's internal tracing system, which tries to export traces to the default OTLP port on IPv6 localhost, even if your OTLP receiver is set to IPv4. This happens independently of your main receiver config and can't be globally disabled with a config flag [source]. The only way to force IPv4 is to make sure every endpoint in your config is explicitly set to an IPv4 address like 0.0.0.0 or 127.0.0.1—avoid localhost, as it can resolve to IPv6 in some environments [source].

Environment variables like OTEL_SDK_DISABLED and OTEL_TRACES_EXPORTER=none help, but they don't affect how Jaeger binds to network interfaces. All critical settings must be pinned in the YAML config. There is no disable_ipv6 flag—it's all about being explicit with your endpoints [source].

For health checks, recent Jaeger versions added better readiness/liveness probe paths, but the config is still picky. If you hit issues, try increasing Jaeger's log level to debug to get more actionable error messages, and use a YAML linter before starting Jaeger to catch syntax problems early [source].

If you want to see working config examples, check the Jaeger repo's cmd/jaeger directory for sample YAML files [source].

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[jkowall]

@ekimminau can you actually post the YAML files for Jaeger? It's hard to read them without formatting the way you pasted them in here. Jaeger is using YAML the same way that Kubernetes and OpenTelemetry use, so we have normalized things with these projects. YAML can be really annoying, but I suggest using an LLM to help with debugging the issues.

[yurishkuro]

I added escaping to the post to render yaml better

[ekimminau]

Trust me, chat gpt-4 stand alone, gpt-4, gpt-5 and claude 4 in github chat within visual code are all being extensively used and this prodct would be IMPOSSIBLE to troubleshoot without.
I would be more than happy to post the yaml without markdown formatting but I honestly dont know how to do it in here. Ill try as code. Nope. That looks even worse. How do I post yaml code blocks?

[jkowall]

You have a lot of basic logic errors here. I can see 3 different metric backends in use, but I am pretty sure you have 1 Prometheus? I do not understand your setup and configuration since you haven't included the prometheus side of things here and you are using metrics in your pipeline.

Your configuration will fail to start because you've used an incorrect name for the Jaeger storage exporter. The exporter that writes to the backends defined in the jaeger_storage extension is simply called jaeger_storage, not jaeger_storage_exporter.

The name you define in the exporters: section is interpreted as the exporter's type. Since jaeger_storage_exporter isn't a valid, built-in exporter type, the service won't launch.

The Fix: You need to rename jaeger_storage_exporter to jaeger_storage in two places: the service.pipelines.traces.exporters list and the top-level exporters definition.

I would drop the metrics part until you get the rest working and add it after, but as I mentioned there are a lot of issues in your config:

Your configuration correctly sets up the Jaeger Query UI to read service metrics from a Prometheus instance. This is what the jaeger_query and jaeger_storage extensions are configured to do:

jaeger_query.storage.metrics points to some_metrics_storage.

jaeger_storage.metric_backends defines some_metrics_storage as a Prometheus endpoint.

However, your configuration does not include a pipeline to receive and write metrics to that Prometheus instance. This is not an error if you are sending your application metrics to Prometheus through another method (e.g., a separate OpenTelemetry Collector or direct Prometheus scraping).

If you intend for this Jaeger instance to also collect and store application metrics, you need to add a metrics pipeline to the service section.