Invalid X-Ray Id with direct gRPC call (ALB + EKS)

[BT-22]

Hello, when using grpcurl to send a gRPC call directly to a service running in our ALB + EKS setup, we can see all the 3 examples at https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-request-tracing.html
ALB appears working correctly.

Our Istio ingress gateway log also indicates the forwarding of header X-Amzn-Trace-Id works correctly.

The service uses aws-opentelemetry-agent.jar for auto-instrumentation.

While the gRPC call executes successfully, we don't have an X-Ray trace:

1. When a gRPC call arrives, the service sees an error message from the aws otel agent:

[otel.javaagent 2022-10-27 17:04:52:358 +0000] [grpc-default-executor-27] ERROR io.opentelemetry.javaagent.shaded.instrumentation.api.internal.ContextPropagationDebug - Unexpected non-root current context found when extracting remote context!

2. I can see from interceptor the header X-Amzn-Trace-Id is indeed passed to the gRPC server.
3. However, it seems like it's not picked up, and a new trace identifier is generated - which is not in X-Ray format.

2022-10-27T17:04:52,359 | [grpc-default-executor-27] | [c.i.a.i.a.i.Interceptor] | INFO | idsvc | traceId: 1-b3667db0-4d3751b9b59a29fe07837643 spanId: f5ce5ccf5e1f9f01 | tenantId: | ******** x-amzn-trace-id from client Self=1-635aba34-538789d31f1b504d3af48d29;Root=1-6350b8d5-93164eabb1697bcd9d3e7ca8

As you can see in the log message, the x-amzn-trace-id has both Root and Self in the right format; but the traceId has the invalid timestamp b3667db0, hence we do not have a trace.

We read and log traceId from AWS-XRAY-TRACE-ID.

---

If the gRPC call is not the very first call, but invoked by another service, in our API Gateway + NLB setup, then the full end-to-end trace does work, and the gRPC call has its segments.

It's the same EKS and the same service pods that are behind both the API Gateway + NLB and the ALB + EKS.
So, we're puzzled with the situation.

• Is it something we're missing in configuring ALB + EKS?
• Or is it something we're missing in using the aws otel agent?
• Or is there something special about gRPC call being the first service?

---

FWIW: In a simplified setup, I run both the service with aws otel agent, and run aws otel collector locally on a PC, without ALB or Istio.
I send the header X-Amzn-Trace-Id in the gRPC request, and it seems like it's also not picked up, and a new trace Id is generated.
But in this local setup, the newly generated trace is a valid X-Ray ID (hence I do see a trace).

Which makes me wonder is there something I'm missing in using the aws otel agent?
Is there something I need to specify to say "use X-Ray ID generator"?

[laurit]

You can ask questions about AWS distro of otel agent at https://github.com/aws-observability/aws-otel-java-instrumentation If you believe this is an issue in the otel agent then please provide a minimal application that reproduces your issue without aws specific stuff.

[BT-22]

Thank you @laurit - I should have posted there indeed. Will do.