Getting following error while pushing JVM metric data to NewRelic

[nimendrak]

I am currently trying to capture JVM metrics of an auto instrumented spring boot application using opentelemetry and push those to newrelic backend. Also the java agent version is 1.15.0

java -javaagent:/lib/opentelemetry-javaagent.jar \ -Dotel.resource.attributes=service.name=${service_name^^} \ -Dotel.resource.attributes=service.instance.id=$pod_uid \ -Dotel.instrumentation.runtime-metrics.experimental-metrics.enabled=true \ -Dotel.exporter.otlp.endpoint=http://otel-collector.observability.svc.cluster.local:4317 \ -jar /lib/myapp-*.jar

But keep getting following error.

ERROR io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter - Failed to export metrics. The request could not be executed. Full error message: Failed to connect to localhost/127.0.0.1:4317

[mateuszrzeszutek]

Hey @nimendrak ,
It looks like you don't have connectivity to http://otel-collector.observability.svc.cluster.local:4317 from the Java app pod. Try running nc -v otel-collector.observability.svc.cluster.local 4317 inside the pod, see what happens; and check your cloud orchestrator (k8s?) network settings, perhaps the collector service is not accessible from the application pod?

[nimendrak]

Yes it does. I have attached the stacktrace here. @mateuszrzeszutek

bash-5.1# ./start.sh 
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[otel.javaagent 2022-09-07 11:13:19:874 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.15.0
Error: Could not find or load main class CXOMS
Caused by: java.lang.ClassNotFoundException: CXOMS
[otel.javaagent 2022-09-07 11:13:28:949 +0000] [OkHttp http://localhost:4317/...] ERROR io.opentelemetry.exporter.internal.grpc.OkHttpGrpcExporter - Failed to export metrics. The request could not be executed. Full error message: Failed to connect to localhost/127.0.0.1:4317
bash-5.1# nc -v otel-collector.observability.svc.cluster.local 4317
otel-collector.observability.svc.cluster.local (10.94.23.238:4317) open

[laurit]

My guess is that the real problem is Error: Could not find or load main class CXOMS which means that you have most likely messed up your jvm arguments. Probably you have an extra space somewhere or are missing quotes so the jvm interprets CXOMS as the main class although it should be part of some other argument. The rest of the arguments that come after the main class are interpreted as arguments to main method which probably also includes your -Dotel.exporter.otlp.endpoint

[nimendrak]

Hey @laurit thanks for the reply. I have changed order of the args and application is working now but the opentelemetry throws the same error.

[laurit]

Is it still attempting to connect to localhost or to otel-collector.observability.svc.cluster.local? If it is still trying to connect to localhost then you probably still have something wrong with the jvm arguments. You could try to set the OTEL_EXPORTER_OTLP_ENDPOINT environment variable, perhaps using env variables is easier for you.

[nimendrak]

Yes, this was an issue related to the jvm arguments. It was all sorted out when I used them as env variables. Thanks for your help @laurit @mateuszrzeszutek