Unable to setup Lambda javaagent Layer in a way that propagates context across SQS

[nic-littlepay]

Our setup

Lambda 1 - manually invoked through function URL it posts a message to the queue
SQS Queue
Lambda 2 - invoked through SQSEvent from the queue

We want a setup that propagates trace_id values from Lambda 1 to Lambda 2, over SQS. For both of these Lambdas we've followed the instructions in the README and are using the Java Agent layer - arn:aws:lambda:${AWS::Region}:184161586896:layer:opentelemetry-javaagent-0_12_0:1 in our AWS::Serverless::Function template. We've also added the env var AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-handler to the template. We are using a log4j2 configuration like <PatternLayout pattern="...trace_id=%X{trace_id}" /> so that we can view trace_id in logs, as per the Log4j MDC instrumentation which we believe is bundled into the auto-instrumentation java agent Lambda layer.

What we are expecting

• invoking Lambda 1 will cause the otel library to generate a trace_id
• Lambda 1 places a message on the queue and that trace_id will be added as SQS message system attribute
• Lambda 2 will process the message and pickup the trace_id value which will be added to the Lambda 2 context and be used in log messages

What we are actually seeing

• otel library generates a trace_id value and we can see this in the logs of Lambda 1
• SQS message is successfully placed on the queue and read by Lambda 2
• (but) Lambda 2 logs show a different trace_id value in logs, the trace_id from Lambda 1 appears to have been lost

What we've tried

Just to test it out we tried adding OTEL_INSTRUMENTATION_AWS_SDK_EXPERIMENTAL_USE_PROPAGATOR_FOR_MESSAGING: true to both Lambdas. This is something we read about here. While using that env var we added some logging to print out SQS message attributes in Lambda 2, and we can see the trace_id value from Lambda 1 appearing in two message attributes:

• Message attribute: X-Amzn-Trace-Id = Root=1-68144a4f-3ef37b2b46dd76234f15a792;Parent=92e7eaa95546c9bf;Sampled=1;Lineage=1:964c8305:0
• Message attribute: traceparent = 00-68144a4f3ef37b2b46dd76234f15a792-92e7eaa95546c9bf-01

But the trace_id=%X{trace_id} ref in the logs is still be resolved to a different trace_id value.
Given we can see this in message attributes I'm proposing that without the OTEL_INSTRUMENTATION_AWS_SDK_EXPERIMENTAL_USE_PROPAGATOR_FOR_MESSAGING flag, the trace_id may still be being passed across as message metadata, although I haven't found a way to spy on this in the code and log it to confirm.

What are we missing that will propagate the context correctly?
Any pointers greatly appreciated 🙏

---

EDIT:
could the behaviour we're seeing be related to #1787