Correct Use of Baggage

[codeforaliving]

Hello, we are currently using Baggage to shuttle a transaction id from our main service and ultimately adding the value as a span attribute in a custom SpanProcessor extension. The extension has been working well, however when we added logic to insert a transaction id into Baggage, the server experienced increased latency and a possible deadlock. We are using OTEL agent v1.12.0.

Is this the correct way to use Baggage?

Main Java Service (insert):
Baggage.current().toBuilder().put("my-transaction-id", "some-transaction-id").build().storeInContext(Context.current()).makeCurrent();

SpanProcessor Extension (get):
... Baggage baggage = Baggage.fromContextOrNull(parentContext); if (baggage != null) { String txId = baggage.getEntryValue("my-transaction-id"); .... }

Thanks in advance!

[trask]

hi @codeforaliving do you have profiles or stack traces showing where the bottleneck/deadlock occurred?

[codeforaliving]

@trask Yes, and there wasn't anything obvious that pointed to Open Telemetry. All I know is that when we add the Baggage logic, latency increases at a steady pace and never recovers.

At the servlet filter level in our Spring Boot app we have this:

...
try {
     // save the transaction id in the context baggage
     final Context c = Context.current();
     c.with(Baggage.builder()
          .put("my-transaction-id", "ABC123")
          .build())
          .makeCurrent();
} catch (Exception ex) {}
...

Then in our custom extension, we have a SpanProcessor that performs the following logic:

...
@Override
public void onStart(Context parentContext, ReadWriteSpan span) {
     final Baggage baggage = Baggage.fromContextOrNull(parentContext);
     if (baggage != null) {
         final String myTransactionId = baggage.getEntryValue("my-transaction-id");
         span.setAttribute("my-transaction-id", myTransactionId);
     }
     ...
}

Essentially, we put something in Baggage in the web app and then extract it in the SpanProcessor in our custom extension. Does this flow make sense? Is there a better way? Thanks in advance. We are using library version 1.12.

[codeforaliving]

@trask I captured a few jstack traces this morning. I always notice the BatchSpanProcessor thread waiting. Not sure if that is an issue yet.

"BatchSpanProcessor_WorkerThread-1" #22 daemon prio=5 os_prio=0 cpu=2584.01ms elapsed=55262.21s tid=0x00007efc9d530000 nid=0x64 waiting on condition  [0x00007efc3b3f3000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.16.1/Native Method)
	- parking to wait for  <0x00000007011e94e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.16.1/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.16.1/AbstractQueuedSynchronizer.java:2123)
	at java.util.concurrent.ArrayBlockingQueue.poll(java.base@11.0.16.1/ArrayBlockingQueue.java:432)
	at io.opentelemetry.sdk.trace.export.BatchSpanProcessor$Worker.run(BatchSpanProcessor.java:246)
	at java.lang.Thread.run(java.base@11.0.16.1/Thread.java:829)

   Locked ownable synchronizers:
	- None

The above stack trace showed up in 3 straight captures.

[laurit]

You always need to close the scope returned from context.makeCurrent(). Failure to do so will result in a memory leak.
There is noting wrong with batch processor thread waiting, if it doesn't have anything to do it will be waiting.