configuration issue with the Prometheus metrics exporter

[mitht]

opentelemetry-javaagent version: 1.33.1
prometheus version: 2.51.1.darwin-amd64
grafana version: 10.4.1

environment variable

OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317;OTEL_INSTRUMENTATION_COMMON_DB_STATEMENT_SANITIZER_ENABLED=false;OTEL_METRICS_EXPORTER=prometheus;OTEL_EXPORTER_PROMETHEUS_HOST=127.0.0.1;OTEL_EXPORTER_PROMETHEUS_PORT=9090

javaagent

-javaagent:/app/opentelemetry-javaagent-1.33.1.jar -Dotel.service.name=my-service -Dotel.instrumentation.jdbc.enabled=true -Dotel.logs.exporter=none

After configuring OTEL_METRICS_EXPORTER=prometheus;OTEL_EXPORTER_PROMETHEUS_HOST=127.0.0.1;OTEL_EXPORTER_PROMETHEUS_PORT=9090, Grafana fails to connect to Prometheus, with the following error message:

Without Prometheus environment variables configured, visit: http://127.0.0.1:9090/graph

With the environment configuration added, OTEL_METRICS_EXPORTER=prometheus;OTEL_EXPORTER_PROMETHEUS_HOST=127.0.0.1;OTEL_EXPORTER_PROMETHEUS_PORT=9090, visit: http://127.0.0.1:9090/graph.

[mitht]

Error message:

ReadObject: expect { or , or } or n, but found #, error found in #1 byte of ...|# HELP http|..., bigger context ...|# HELP http_server_duration_milliseconds The durati|... - There was an error returned querying the Prometheus API.

[trask]

hi @mitht, is there any exception stack trace logged in the application that's running the java agent?

if not, can you try to repro using the OTel Java SDK + Prometheus exporter directly to help narrow down the issue?

[mitht]

Prometheus and Java program logs are normal

After adding the environment configuration OTEL_METRICS_EXPORTER=prometheus;OTEL_EXPORTER_PROMETHEUS_HOST=127.0.0.1;OTEL_EXPORTER_PROMETHEUS_PORT=9090 and starting the Java program, Grafana cannot connect to Prometheus.

grafana logs:

{"@Level":"warn","@message":"Failed to get prometheus buildinfo","@timestamp":"2024-04-17T10:56:32.175530+08:00","err":"failed to unmarshal JSON: invalid character '#' looking for beginning of value","logger":"tsdb.prometheus"}
{"@Level":"warn","@message":"Failed to get prometheus heuristics","@timestamp":"2024-04-17T10:56:32.175623+08:00","err":"failed to get buildinfo: failed to unmarshal JSON: invalid character '#' looking for beginning of value","logger":"tsdb.prometheus"}
INFO [04-17|10:56:32] Request Completed logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/uid/edisdw2bm1b0gd/health status=400 remote_addr=127.0.0.1 time_ms=75 duration=75.867773ms size=261 referer=http://127.0.0.1:3000/connections/datasources/edit/edisdw2bm1b0gd handler=/api/datasources/uid/:uid/health status_source=server
INFO [04-17|10:56:40] Completed cleanup jobs logger=cleanup duration=46.949668ms

After removing these configurations, access is normal: OTEL_METRICS_EXPORTER=prometheus; OTEL_EXPORTER_PROMETHEUS_HOST=127.0.0.1; OTEL_EXPORTER_PROMETHEUS_PORT=9090

[laurit]

cc @zeitlinger

[mitht]

The issue is resolved by adding the following configuration to Prometheus.
The port 9090 should not be used.

• job_name: "otelcol"
  static_configs:
  - targets: ["localhost:17890"]