Multiple trace contexts are getting generated for single transaction

[srinivas-bode]

Hi All,

I am using opentelemetry java agent v1.5.0 for traces and spans for OSB(Oracle service Bus). Please find my configurations below.
otel.metrics.exporter=none
otel.instrumentation.common.default-enabled=false
otel.instrumentation.methods.include=com.bea.wli.sb.transports.http.HttpOutboundMessageContext[send,getResponse,setRequestMetaData,setRequestProperty,setRequestPayload,getResponsePayload,scheduleResponseWork,buildEffectiveUrl,getHttpUrlConnectionFactory,resolve,getResponse,isReturnPayload,handleResponse];com.bea.wli.sb.transports.http.HttpOutboundMessageContext$PostHttpResponseWork[run]
otel.instrumentation.cxf.enabled=true
otel.instrumentation.apache-httpclient.enabled=true
otel.instrumentation.java-http-client.enabled=true
otel.instrumentation.http-url-connection.enabled=true
otel.instrumentation.jaxrs-client.enabled=true
otel.instrumentation.jaxrs.enabled=true
otel.instrumentation.jaxws.enabled=true
otel.instrumentation.grpc.enabled=true
otel.instrumentation.apache-httpasyncclient.enabled=true
otel.instrumentation.async-http-client.enabled=true
otel.instrumentation.guava.enabled=true
otel.instrumentation.servlet.enabled=true
otel.instrumentation.okhttp.enabled=true
otel.instrumentation.spring-core.enabled=true
otel.instrumentation.spring-webmvc.enabled=true
otel.instrumentation.spring-ws.enabled=true
otel.instrumentation.rmi.enabled=true
otel.instrumentation.jdbc.enabled=true
otel.instrumentation.jms.enabled=true
otel.instrumentation.netty.enabled=true
otel.instrumentation.reactor-netty.enabled=true
otel.instrumentation.google-http-client.enabled=true
otel.instrumentation.restlet.enabled=true
otel.instrumentation.reactor.enabled=true
otel.instrumentation.metro.enabled=true
otel.instrumentation.grizzly.enabled=true
otel.resource.attributes=service.name=osb
otel.traces.exporter=otlp
otel.javaagent.debug=true
otel.exporter.otlp.protocol=http/protobuf

Issue:
In one of the case when i invoke a webservice which makes two calls, one for service A and other for Service B. For this there are two traces getting generated, one for each service call. Is there a way to get both of them under a single traceid or correlate both so i will have the entire flow. Manual instrumentation is not possible as its an enterprise product and i cannot make changes to source code.

Thanks in advance.

[jkwatson]

Moving this over to the instrumentation repository

[trask]

hi @srinivas-bode,

the first thing I'd try is to use a recent Java agent version (1.5.0 is nearing 3 years old), and also try without otel.instrumentation.common.default-enabled=false in case you're disabling some instrumentation which is needed for context propagation

if that doesn't help, we'd probably need a repro to help track down where context propagation is being lost

[srinivas-bode]

Hi @trask ,

Yeah i was using that because when i am instrumenting everything i am getting below error while instrumenting the metro library.
We are using Oracle Service Bus. We are trying to explore opentelemetry java agent for collecting traces from the application. We we do so we are getting the below error.
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
soapenv:Body
soapenv:Fault
soapenv:Server
OSB-382000: Not yet supported!

<con:stack-trace xmlns:con="http://www.bea.com/wli/sb/context">com.bea.wli.sb.service.handlerchain.HandlerException: Not yet supported!
at com.bea.wli.sb.service.disi.handlerchain.handlers.InboundDISIHandler.dispatch(InboundDISIHandler.java:144)
at com.bea.wli.sb.service.handlerchain.handlers.AbstractHandler.dispatch(AbstractHandler.java:129)
at com.bea.wli.sb.service.disi.handlerchain.handlers.InboundWssPhase1DISIHandler.dispatch(InboundWssPhase1DISIHandler.java:100)
at com.bea.wli.sb.service.handlerchain.handlers.AbstractHandler.dispatch(AbstractHandler.java:129)
at com.bea.wli.sb.service.handlerchain.handlers.InboundMessageContentHandler.dispatch(InboundMessageContentHandler.java:170)
at com.bea.wli.sb.service.handlerchain.handlers.AbstractHandler.dispatch(AbstractHandler.java:129)
at com.bea.wli.sb.service.handlerchain.handlers.CheckAccessControl.dispatch(CheckAccessControl.java:73)
at com.bea.wli.sb.service.handlerchain.handlers.AbstractHandler.dispatch(AbstractHandler.java:129)
at com.bea.wli.sb.service.handlerchain.handlers.UpdateInboundTransportStatistics.dispatch(UpdateInboundTransportStatistics.java:64)
at com.bea.wli.sb.service.handlerchain.handlers.AbstractHandler.dispatch(AbstractHandler.java:129)
at com.bea.wli.sb.service.handlerchain.handlers.AbstractHandler.dispatch(AbstractHandler.java:129)
at com.bea.wli.sb.service.handlerchain.handlers.DefaultInboundErrorHandler.dispatch(DefaultInboundErrorHandler.java:44)
at com.bea.wli.sb.service.handlerchain.DefaultInboundHandlerChain.dispatch(DefaultInboundHandlerChain.java:135)
at com.bea.wli.sb.transports.TransportDispatcherClient.dispatch(TransportDispatcherClient.java:96)
at com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:330)
at com.bea.wli.sb.transports.http.generic.RequestHelperBase.invokePipeline(RequestHelperBase.java:253)
at com.bea.wli.sb.transports.http.wls.HttpTransportServlet$RequestHelperWLS.invokePipeline(HttpTransportServlet.java:257)
at com.bea.wli.sb.transports.http.generic.RequestHelperBase$1.run(RequestHelperBase.java:220)
at com.bea.wli.sb.transports.http.generic.RequestHelperBase$1.run(RequestHelperBase.java:216)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:386)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:163)
at com.bea.wli.sb.util.security.SecurityUtils.executeAs(SecurityUtils.java:102)
at com.bea.wli.sb.transports.http.generic.RequestHelperBase.securedInvoke(RequestHelperBase.java:214)
at com.bea.wli.sb.transports.http.generic.RequestHelperBase.service(RequestHelperBase.java:142)
at com.bea.wli.sb.transports.http.wls.HttpTransportServlet.service(HttpTransportServlet.java:150)
at weblogic.servlet.FutureResponseServlet.service(FutureResponseServlet.java:24)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:295)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:353)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:248)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3800)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3766)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:344)
at weblogic.security.service.SecurityManager.runAsForUserCode(SecurityManager.java:197)
at weblogic.servlet.provider.WlsSecurityProvider.runAsForUserCode(WlsSecurityProvider.java:203)
at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:71)
at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2454)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2302)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2280)
at weblogic.servlet.internal.ServletRequestImpl.runInternal(ServletRequestImpl.java:1739)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1699)
at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:272)
at weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:352)
at weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:337)
at weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:57)
at weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)
at weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:655)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:420)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:360)
Caused by: java.lang.UnsupportedOperationException: Not yet supported!
at com.bea.wli.sb.security.wss.wsm.CommonWsmHandler$OSBWSMHttpRequest.getServletPath(CommonWsmHandler.java:728)
at io.opentelemetry.javaagent.instrumentation.metro.MetroServerSpanNaming.updateServerSpanName(MetroServerSpanNaming.java:32)
at io.opentelemetry.javaagent.instrumentation.metro.MetroHelper.start(MetroHelper.java:27)
at io.opentelemetry.javaagent.instrumentation.metro.TracingTube.processRequest(TracingTube.java:37)
at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:1136)
at com.sun.xml.ws.api.pipe.Fiber.access$100(Fiber.java:127)
at com.sun.xml.ws.api.pipe.Fiber$InterceptorHandler.execute(Fiber.java:965)
at com.sun.xml.ws.api.pipe.Fiber$InterceptorHandler.execute(Fiber.java:939)
at com.oracle.webservices.impl.disi.service.ServiceRequestTransportProcessor$Receiver$2.execute(ServiceRequestTransportProcessor.java:106)
at com.sun.xml.ws.api.pipe.Fiber$InterceptorHandler.execute(Fiber.java:969)
at com.sun.xml.ws.api.pipe.Fiber$InterceptorHandler.invoke(Fiber.java:958)
at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:1054)
at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:1019)
at com.sun.xml.ws.api.pipe.Fiber.run(Fiber.java:813)
at com.sun.xml.ws.api.server.ThreadLocalContainerResolver$2$1.run(ThreadLocalContainerResolver.java:112)
at com.oracle.webservices.impl.disi.service.DISIEndpointImpl$1.execute(DISIEndpointImpl.java:141)
at com.sun.xml.ws.api.server.ThreadLocalContainerResolver$2.execute(ThreadLocalContainerResolver.java:107)
at com.sun.xml.ws.api.pipe.Engine.addRunnable(Engine.java:96)
at com.sun.xml.ws.api.pipe.Fiber.start(Fiber.java:424)
at com.sun.xml.ws.server.WSEndpointImpl.processAsync(WSEndpointImpl.java:368)
at com.sun.xml.ws.server.WSEndpointImpl.schedule(WSEndpointImpl.java:322)
at com.sun.xml.ws.metro.api.config.management.ManagedEndpoint.schedule(ManagedEndpoint.java:230)
at com.oracle.webservices.impl.disi.service.ServiceRequestTransportProcessor$Receiver.run(ServiceRequestTransportProcessor.java:92)
at com.oracle.webservices.impl.disi.service.ServiceRequestTransportProcessor.request(ServiceRequestTransportProcessor.java:53)
at com.oracle.webservices.impl.disi.service.DISIAdapter$DISIToolkit.invoke(DISIAdapter.java:315)
at com.oracle.webservices.impl.disi.service.DISIAdapter$2.request(DISIAdapter.java:142)
at com.oracle.webservices.impl.disi.service.DISIAdapter$2.request(DISIAdapter.java:105)
at com.bea.wli.sb.service.disi.handlerchain.handlers.InboundDISIHandler$1.run(InboundDISIHandler.java:113)
at java.security.AccessController.doPrivileged(Native Method)
at oracle.security.jps.util.JpsSubject.doAs(JpsSubject.java:207)
at com.bea.wli.sb.service.disi.handlerchain.handlers.InboundDISIHandler.dispatch(InboundDISIHandler.java:110)
... 53 more</con:stack-trace>

</soapenv:Fault>
</soapenv:Body>
</soapenv:Envelope>

So i suppressed the instrumentation of the library, i am not getting the issue anymore but as i told earlier the issue i am facing now is with trace propagation.

I already raised an issue for this below is the link to that
#10986