Unable to send metrics to OpenTelemetry collector - Failed to connect to localhost :4318

[rapar8]

Application is using java otel agent

FROM maven:3.9.3-eclipse-temurin-17-alpine as service-layer-build
WORKDIR /service-layer
COPY . /service-layer
RUN mvn clean package -DskipTests -U --no-transfer-progress && rm -rf /service-layer/.git

FROM eclipse-temurin:17.0.12_7-jre-alpine

WORKDIR /service-layer
RUN apk --no-cache add curl
RUN curl -L https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar --output otel-javaagent.jar
CMD ["java", "-javaagent:otel-javaagent.jar", "-jar","service-layer.jar"]

Docker Compose.yaml

services:
  service-layer:
    build: .
    image: service-layer:latest
    container_name: service-layer
    hostname: service-layer
    depends_on:
      - otel-collector
    ports:
      - "8080:8080"
    networks:
      - inst-java-apps-using-otel
    environment:
      - OTEL_TRACES_EXPORTER=otlp
      - OTEL_METRICS_EXPORTER=otlp
      - OTEL_LOGS_EXPORTER=otlp
      - OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
      - OTEL_INSTRUMENTATION_HTTP_CLIENT_EMIT_EXPERIMENTAL_TELEMETRY=true
      - OTEL_INSTRUMENTATION_HTTP_SERVER_EMIT_EXPERIMENTAL_TELEMETRY=true
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317      <-- **tried 4318**
      - OTEL_TRACES_SAMPLER=always_on
      - OTEL_IMR_EXPORT_INTERVAL=5000
      - OTEL_SERVICE_NAME=omsvclyr
      - OTEL_METRIC_EXPORT_INTERVAL=5000
      - OTEL_RESOURCE_ATTRIBUTES="job=job,application=omsvclyr,service.name=service-layer,service.version=1.0,deployment.environment=development"
    healthcheck:
      interval: 5s
      retries: 10
      test: curl --write-out 'HTTP %{http_code}' --fail --silent --output /dev/null http://localhost:8080/test/env

  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    restart: always
    container_name: otel-collector
    hostname: otel-collector
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC receiver
      - "4318:4318"   # OTLP http receiver
    networks:
      - inst-java-apps-using-otel
    extra_hosts:
      - "host.docker.internal:host-gateway"

  tempo:
    image: grafana/tempo:1.5.0
    command: [ "-search.enabled=true", "-config.file=/etc/tempo.yaml" ]
    container_name: tempo
    hostname: tempo
    volumes:
      - ./o11y-backend/tempo-config.yaml:/etc/tempo.yaml
      - ./o11y-backend/tempo-overrides.yaml:/etc/overrides.yaml
      - ./tempo-data:/var/lib/tempo
    ports:
      - "3200:3200"
      - "4320:4320"
    networks:
      - inst-java-apps-using-otel
    healthcheck:
      interval: 5s
      retries: 10
      test: wget --no-verbose --tries=1 --spider http://localhost:3200/status || exit 1
#
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    hostname: prometheus
    command:
      - --config.file=/etc/prometheus.yaml
      - --web.enable-remote-write-receiver
      - --enable-feature=exemplar-storage
    volumes:
      - ./o11y-backend/prometheus.yaml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
    networks:
      - inst-java-apps-using-otel
    healthcheck:
      interval: 5s
      retries: 10
      test: wget --no-verbose --tries=1 --spider http://localhost:9090/status || exit 1

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    hostname: grafana 
#    depends_on:
#      tempo:
#        condition: service_healthy
#      prometheus:
#        condition: service_healthy
    volumes:
      - ./o11y-backend/grafana-bootstrap.ini:/etc/grafana/grafana.ini
      - ./o11y-backend/grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_DISABLE_LOGIN_FORM=true
      - GF_LOG_MODE=console file
      - GF_LOG_LEVEL=info
      - GF_PATHS_LOGS="/var/log/grafana"
    ports:
      - "3000:3000"
    networks:
      - inst-java-apps-using-otel
    healthcheck:
      interval: 5s
      retries: 10
      test: wget --no-verbose --tries=1 --spider http://localhost:3000 || exit 1

networks:
  inst-java-apps-using-otel:
    driver: bridge
    name: inst-java-apps-using-otel

otel-collector-config.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317
      http:
        endpoint: localhost:4318

exporters:
  prometheus:
    endpoint: localhost:9090
    namespace: default
  otlp:
    endpoint: localhost:4317
    tls:
      insecure: true
  logging:
    loglevel: debug

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheus, logging]
    traces:
      receivers: [otlp]
      exporters: [logging, otlp]
  telemetry:
    logs:
      level: debug

Error:

 [OkHttp http://localhost:4318/...] ERROR io.opentelemetry.exporter.internal.http.HttpExporter - Failed to export metrics|logs. The request could not be executed. Full error message: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:4318
service-layer  | java.net.ConnectException: Failed to connect to localhost/[0:0:0:0:0:0:0:1]:4318
... Connection refused

Docker ps

otel/opentelemetry-collector-contrib:latest "/otelcol-contrib --…" 24 minutes ago Up 24 minutes 0.0.0.0:4317->4317/tcp, :::4317->4317/tcp, 55678-55679/tcp otel-collector

Docker inspect otel-collector

"PortBindings": {
"4317/tcp": [
{
"HostIp": "",
"HostPort": "4317"
}
]
},

...
"NetworkSettings": {
"Bridge": "",
"SandboxID": "ecbb4197d06eaa14facbfc09415ff87db48506893bd7aa46645deff59e8a70d2",
"SandboxKey": "/var/run/docker/netns/ecbb4197d06e",
"Ports": {
"4317/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "4317"
},
{
"HostIp": "::",
"HostPort": "4317"
}
],
...
"IPAddress": "172.21.0.2",

I tried to access

• localhost:4318
• 172.21.0.2:4318

[jaydeluca]

Discussion has continued in #12180

A few notes from discussion in that issue:

• The address for the collector when called from another docker container should use the container name (otel-collector) instead of localhost
• The collector configuration endpoints should match the route being used by the application (example; otel-collector:4137 for http/protobuf)
• The collector config needs to have pipelines setup that map to the otlp receivers for each signal, or unneeded signals should be turned off (example: OTEL_LOGS_EXPORTER=none)