Skip to content
Open
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
796b57e
Implement APIGW Inferred Proxy Spans (#8336)
jordan-wong May 7, 2025
e8c2baf
feat(serverless): Fix gateway inferred span design
PerfectSlayer May 12, 2025
737e783
init commit of instrumentations
zarirhamza Aug 26, 2025
c9a2e21
fix tests
zarirhamza Aug 26, 2025
a0ecac7
round 2 of context migration
zarirhamza Aug 26, 2025
2ef3dc7
netty changes
zarirhamza Aug 26, 2025
a14848f
tomcat changes
zarirhamza Aug 26, 2025
112ec20
liberty changes
zarirhamza Aug 26, 2025
909d174
store span
zarirhamza Aug 26, 2025
c883e8d
jetty
zarirhamza Aug 26, 2025
b847401
grizzly
zarirhamza Aug 26, 2025
bebe637
Merge branch 'master' into zarir/context-api-finish
zarirhamza Aug 26, 2025
f76415f
fix grizzly instrumentation
zarirhamza Aug 26, 2025
8f4a6ee
Merge branch 'master' into zarir/context-api-finish
zarirhamza Aug 27, 2025
762c233
remove serverless
zarirhamza Aug 27, 2025
c8c9ece
fixes part 1
zarirhamza Aug 27, 2025
f715d62
fix undertow tests
zarirhamza Aug 27, 2025
7fe9c70
remove span within request
zarirhamza Aug 27, 2025
0f0f228
fix tests
zarirhamza Aug 27, 2025
ecc5049
fix tests
zarirhamza Aug 27, 2025
4bd005d
implement fixes part 1
zarirhamza Aug 28, 2025
9b4d887
fix tests
zarirhamza Aug 28, 2025
a3bb790
merge main fixing merge conflicts
zarirhamza Oct 2, 2025
ee71b23
address jetty comments
zarirhamza Oct 4, 2025
dd4c4c0
Merge branch 'master' into zarir/context-api-finish
zarirhamza Oct 4, 2025
7fd52d8
Merge branch 'master' into zarir/context-api-finish
zarirhamza Oct 8, 2025
2c27431
use context scope
zarirhamza Oct 25, 2025
6e950a0
fix netty tests
zarirhamza Oct 25, 2025
5f20aee
Merge branch 'master' into zarir/context-api-finish
zarirhamza Oct 25, 2025
fd11e7a
fix baggage context usage
zarirhamza Oct 30, 2025
0c4ab54
Merge branch 'master' into zarir/context-api-finish
zarirhamza Oct 30, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
import static datadog.trace.api.cache.RadixTreeCache.UNSET_PORT;
import static datadog.trace.bootstrap.instrumentation.java.net.HostNameResolver.hostName;

import datadog.context.Context;
import datadog.context.ContextScope;
import datadog.trace.api.Config;
import datadog.trace.api.DDTags;
Expand Down Expand Up @@ -76,15 +77,19 @@ public AgentSpan afterStart(final AgentSpan span) {
return span;
}

public AgentScope beforeFinish(final AgentScope scope) {
beforeFinish(scope.span());
public ContextScope beforeFinish(final ContextScope scope) {
beforeFinish(scope.context());
return scope;
}

public AgentSpan beforeFinish(final AgentSpan span) {
return span;
}
Comment on lines 85 to 87
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❔ question: ‏Can this one be removed then? It will prevent other decorator to override it and never being called.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cannot, WebSocketDecorator and its related instrumentation still call it. Needs to be a separate mini-migration

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that the only one blocker? Because the WebSocketDecorator is calling the empty BaseDecorator.beforeFinish() and has no override... So technically, the call is doing nothing :/
So we can move it from BaseDecorator to WebSocketDecorator and migrate it in a second pass.

@amarziali What's the effort to migrate the WebSocket instrumentation to context rather than span?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another place where this is called is within the Mule Instrumentation as well via the MuleDecorator in mule-4.

Moved the empty call to WebSocketDecorator and MuleDecorator thus allowing me to remove the function from BaseDecorator.

I assume the actual span -> context functionality will be taken care of later for both WebSockets and Mule.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I take it back, I'm not sure how the tests aren't catching it but the ClientDecorator also uses beforeFinish(span) via the BaseDecorator. This seems to indicate that we need to invest time into looking through all decorators and confirming their reliance on beforeFinish. To save time, for now we can focus just on HttpServerDecorator related instrumentation


public Context beforeFinish(final Context context) {
return context;
}

public AgentScope onError(final AgentScope scope, final Throwable throwable) {
onError(scope.span(), throwable);
return scope;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ public abstract class HttpServerDecorator<REQUEST, CONNECTION, RESPONSE, REQUEST
private static final Logger log = LoggerFactory.getLogger(HttpServerDecorator.class);
private static final int UNSET_PORT = 0;

public static final String DD_SPAN_ATTRIBUTE = "datadog.span";
public static final String DD_CONTEXT_ATTRIBUTE = "datadog.context";
public static final String DD_DISPATCH_SPAN_ATTRIBUTE = "datadog.span.dispatch";
public static final String DD_RUM_INJECTED = "datadog.rum.injected";
public static final String DD_FIN_DISP_LIST_SPAN_ATTRIBUTE =
Expand Down Expand Up @@ -147,17 +147,18 @@ public Context startSpan(REQUEST_CARRIER carrier, Context context) {
instrumentationNames != null && instrumentationNames.length > 0
? instrumentationNames[0]
: DEFAULT_INSTRUMENTATION_NAME;
AgentSpanContext.Extracted extracted = callIGCallbackStart(getExtractedSpanContext(context));
AgentSpanContext extracted = getExtractedSpanContext(context);
// Call IG callbacks
extracted = callIGCallbackStart(extracted);
AgentSpan span =
tracer().startSpan(instrumentationName, spanName(), extracted).setMeasured(true);
// Apply RequestBlockingAction if any
Flow<Void> flow = callIGCallbackRequestHeaders(span, carrier);
if (flow.getAction() instanceof RequestBlockingAction) {
span.setRequestBlockingAction((RequestBlockingAction) flow.getAction());
}
AgentPropagation.ContextVisitor<REQUEST_CARRIER> getter = getter();
if (null != carrier && null != getter) {
tracer().getDataStreamsMonitoring().setCheckpoint(span, forHttpServer());
}
// DSM Checkpoint
tracer().getDataStreamsMonitoring().setCheckpoint(span, forHttpServer());
return context.with(span);
}

Expand Down Expand Up @@ -381,8 +382,7 @@ public AgentSpan onResponse(final AgentSpan span, final RESPONSE response) {
return span;
}

private AgentSpanContext.Extracted callIGCallbackStart(
@Nullable final AgentSpanContext.Extracted extracted) {
private AgentSpanContext callIGCallbackStart(@Nullable final AgentSpanContext extracted) {
AgentTracer.TracerAPI tracer = tracer();
Supplier<Flow<Object>> startedCbAppSec =
tracer.getCallbackProvider(RequestContextSlot.APPSEC).getCallback(EVENTS.requestStarted());
Expand Down Expand Up @@ -517,9 +517,11 @@ private Flow<Void> callIGCallbackURI(
}

@Override
public AgentSpan beforeFinish(AgentSpan span) {
public Context beforeFinish(Context context) {
AgentSpan span = AgentSpan.fromContext(context);
onRequestEndForInstrumentationGateway(span);
return super.beforeFinish(span);

return super.beforeFinish(context);
}

private void onRequestEndForInstrumentationGateway(@Nonnull final AgentSpan span) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,15 +38,15 @@ public Future<HttpResponse> apply(final HttpRequest request) {
request.discardEntityBytes(materializer);
HttpResponse response = BlockingResponseHelper.maybeCreateBlockingResponse(rba, request);
span.getRequestContext().getTraceSegment().effectivelyBlocked();
DatadogWrapperHelper.finishSpan(span, response);
DatadogWrapperHelper.finishSpan(scope.context(), response);
return FastFuture$.MODULE$.<HttpResponse>successful().apply(response);
}

try {
futureResponse = userHandler.apply(request);
} catch (final Throwable t) {
scope.close();
DatadogWrapperHelper.finishSpan(span, t);
DatadogWrapperHelper.finishSpan(scope.context(), t);
throw t;
}

Expand All @@ -67,14 +67,14 @@ public HttpResponse apply(HttpResponse response) {
response = newResponse;
}

DatadogWrapperHelper.finishSpan(span, response);
DatadogWrapperHelper.finishSpan(scope.context(), response);
return response;
}
},
new AbstractFunction1<Throwable, Throwable>() {
@Override
public Throwable apply(final Throwable t) {
DatadogWrapperHelper.finishSpan(span, t);
DatadogWrapperHelper.finishSpan(scope.context(), t);
return t;
}
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ public void onPush() throws Exception {
skipNextPull[0] = true;
requestContext.getTraceSegment().effectivelyBlocked();
emit(responseOutlet, response);
DatadogWrapperHelper.finishSpan(span, response);
DatadogWrapperHelper.finishSpan(scope.context(), response);
pull(requestInlet);
scope.close();
return;
Expand Down Expand Up @@ -142,7 +142,7 @@ public void onPush() throws Exception {
response.discardEntityBytes(materializer());
response = newResponse;
}
DatadogWrapperHelper.finishSpan(span, response);
DatadogWrapperHelper.finishSpan(scope.context(), response);
// Check if the active span matches the scope from when the request came in,
// and close it. If it's not, then it will be cleaned up actor message
// processing instrumentation that drives this state machine
Expand Down Expand Up @@ -172,7 +172,7 @@ public void onUpstreamFailure(final Throwable ex) throws Exception {
if (scope != null) {
// Mark the span as failed
AgentSpan span = fromContext(scope.context());
DatadogWrapperHelper.finishSpan(span, ex);
DatadogWrapperHelper.finishSpan(scope.context(), ex);
}
// We will not receive any more responses from the user code, so clean up any
// remaining spans
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,17 +20,19 @@ public static ContextScope createSpan(final HttpRequest request) {
return context.attach();
}

public static void finishSpan(final AgentSpan span, final HttpResponse response) {
public static void finishSpan(final Context context, final HttpResponse response) {
final AgentSpan span = fromContext(context);
DECORATE.onResponse(span, response);
DECORATE.beforeFinish(span);
DECORATE.beforeFinish(context);

span.finish();
}

public static void finishSpan(final AgentSpan span, final Throwable t) {
public static void finishSpan(final Context context, final Throwable t) {
final AgentSpan span = fromContext(context);
DECORATE.onError(span, t);
span.setHttpStatusCode(500);
DECORATE.beforeFinish(span);
DECORATE.beforeFinish(context);

span.finish();
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ public static void onExit(
if (throwable != null) {
DECORATE.onError(span, throwable);
}
DECORATE.beforeFinish(span);
DECORATE.beforeFinish(scope.context());
} finally {
scope.close();
span.finish();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ public static void onExit(
if (throwable != null) {
DECORATE.onError(span, throwable);
}
DECORATE.beforeFinish(span);
DECORATE.beforeFinish(scope.context());
} finally {
scope.close();
span.finish();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ public static void methodExit(
final AgentSpan span = fromContext(scope.context());
DECORATE.onError(span, throwable);
DECORATE.onResponse(span, response);
DECORATE.beforeFinish(span);
DECORATE.beforeFinish(scope.context());
scope.close();
span.finish();
}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package datadog.trace.instrumentation.cxf;

import com.google.auto.service.AutoService;
import datadog.context.Context;
import datadog.trace.agent.tooling.Instrumenter;
import datadog.trace.agent.tooling.InstrumenterModule;
import datadog.trace.agent.tooling.bytebuddy.matcher.ClassLoaderMatchers;
Expand All @@ -9,6 +10,7 @@
import datadog.trace.bootstrap.instrumentation.api.AgentScope;
import datadog.trace.bootstrap.instrumentation.api.AgentSpan;
import datadog.trace.bootstrap.instrumentation.api.AgentTracer;
import datadog.trace.bootstrap.instrumentation.api.Java8BytecodeBridge;
import datadog.trace.bootstrap.instrumentation.decorator.HttpServerDecorator;
import net.bytebuddy.asm.Advice;
import net.bytebuddy.description.type.TypeDescription;
Expand Down Expand Up @@ -59,11 +61,16 @@ public static AgentScope beforeInvoke(@Advice.Argument(0) final Exchange exchang
if (exchange == null || exchange.getInMessage() == null || AgentTracer.activeSpan() != null) {
return null;
}
final Object span =
final Object contextObj =
ServletHelper.getServletRequestAttribute(
exchange.getInMessage().get("HTTP.REQUEST"), HttpServerDecorator.DD_SPAN_ATTRIBUTE);
if (span instanceof AgentSpan) {
return AgentTracer.activateSpan((AgentSpan) span);
exchange.getInMessage().get("HTTP.REQUEST"),
HttpServerDecorator.DD_CONTEXT_ATTRIBUTE);
if (contextObj instanceof Context) {
Context context = (Context) contextObj;
AgentSpan span = Java8BytecodeBridge.spanFromContext(context);
if (span != null) {
return AgentTracer.activateSpan(span);
}
}
return null;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ public static void setupCallback(
final AgentSpan span = scope.span();
if (throwable != null) {
DECORATE.onError(span, throwable);
DECORATE.beforeFinish(span);
DECORATE.beforeFinish(scope.context());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same thing here, may need to migrate to ContextScope.

span.finish();
scope.close();
return;
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
package datadog.trace.instrumentation.grizzly;

import static datadog.trace.bootstrap.instrumentation.api.Java8BytecodeBridge.spanFromContext;
import static datadog.trace.instrumentation.grizzly.GrizzlyDecorator.DECORATE;

import datadog.appsec.api.blocking.BlockingContentType;
import datadog.context.Context;
import datadog.trace.api.gateway.Flow;
import datadog.trace.bootstrap.blocking.BlockingActionHelper;
import datadog.trace.bootstrap.instrumentation.api.AgentSpan;
Expand Down Expand Up @@ -35,14 +37,14 @@ public class GrizzlyBlockingHelper {
private GrizzlyBlockingHelper() {}

public static boolean block(
Request request, Response response, Flow.Action.RequestBlockingAction rba, AgentSpan span) {
Request request, Response response, Flow.Action.RequestBlockingAction rba, Context context) {
return block(
request,
response,
rba.getStatusCode(),
rba.getBlockingContentType(),
rba.getExtraHeaders(),
span);
context);
}

public static boolean block(
Expand All @@ -51,11 +53,12 @@ public static boolean block(
int statusCode,
BlockingContentType bct,
Map<String, String> extraHeaders,
AgentSpan span) {
Context context) {
if (GET_OUTPUT_STREAM == null) {
return false;
}

AgentSpan span = spanFromContext(context);
try {
OutputStream os = (OutputStream) GET_OUTPUT_STREAM.invoke(response);
response.setStatus(BlockingActionHelper.getHttpCode(statusCode));
Expand All @@ -76,13 +79,17 @@ public static boolean block(
os.close();
response.finish();

span.getRequestContext().getTraceSegment().effectivelyBlocked();
if (span != null) {
span.getRequestContext().getTraceSegment().effectivelyBlocked();
}
SpanClosingListener.LISTENER.onAfterService(request);
} catch (Throwable e) {
log.info("Error committing blocking response", e);
DECORATE.onError(span, e);
DECORATE.beforeFinish(span);
span.finish();
if (span != null) {
DECORATE.onError(span, e);
DECORATE.beforeFinish(context);
span.finish();
}
}

return true;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

import static datadog.trace.agent.tooling.bytebuddy.matcher.NameMatchers.named;
import static datadog.trace.bootstrap.instrumentation.api.Java8BytecodeBridge.spanFromContext;
import static datadog.trace.bootstrap.instrumentation.decorator.HttpServerDecorator.DD_SPAN_ATTRIBUTE;
import static datadog.trace.bootstrap.instrumentation.decorator.HttpServerDecorator.DD_CONTEXT_ATTRIBUTE;
import static datadog.trace.instrumentation.grizzly.GrizzlyDecorator.DECORATE;
import static net.bytebuddy.matcher.ElementMatchers.isMethod;
import static net.bytebuddy.matcher.ElementMatchers.takesArgument;
Expand Down Expand Up @@ -68,7 +68,7 @@ public static class HandleAdvice {
@Advice.Local("contextScope") ContextScope scope,
@Advice.Argument(0) final Request request,
@Advice.Argument(1) final Response response) {
if (request.getAttribute(DD_SPAN_ATTRIBUTE) != null) {
if (request.getAttribute(DD_CONTEXT_ATTRIBUTE) != null) {
return false;
}

Expand All @@ -80,14 +80,14 @@ public static class HandleAdvice {

scope = context.attach();

request.setAttribute(DD_SPAN_ATTRIBUTE, span);
request.setAttribute(DD_CONTEXT_ATTRIBUTE, context);
request.setAttribute(
CorrelationIdentifier.getTraceIdKey(), CorrelationIdentifier.getTraceId());
request.setAttribute(CorrelationIdentifier.getSpanIdKey(), CorrelationIdentifier.getSpanId());

Flow.Action.RequestBlockingAction rba = span.getRequestBlockingAction();
if (rba != null) {
boolean success = GrizzlyBlockingHelper.block(request, response, rba, span);
boolean success = GrizzlyBlockingHelper.block(request, response, rba, context);
if (success) {
return true; /* skip body */
}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
package datadog.trace.instrumentation.grizzly;

import static datadog.trace.bootstrap.instrumentation.decorator.HttpServerDecorator.DD_SPAN_ATTRIBUTE;
import static datadog.trace.bootstrap.instrumentation.api.Java8BytecodeBridge.spanFromContext;
import static datadog.trace.bootstrap.instrumentation.decorator.HttpServerDecorator.DD_CONTEXT_ATTRIBUTE;
import static datadog.trace.instrumentation.grizzly.GrizzlyDecorator.DECORATE;

import datadog.context.Context;
import datadog.trace.bootstrap.instrumentation.api.AgentSpan;
import org.glassfish.grizzly.http.server.AfterServiceListener;
import org.glassfish.grizzly.http.server.Request;
Expand All @@ -12,13 +14,17 @@ public class SpanClosingListener implements AfterServiceListener {

@Override
public void onAfterService(final Request request) {
final Object spanAttr = request.getAttribute(DD_SPAN_ATTRIBUTE);
if (spanAttr instanceof AgentSpan) {
request.removeAttribute(DD_SPAN_ATTRIBUTE);
final AgentSpan span = (AgentSpan) spanAttr;
DECORATE.onResponse(span, request.getResponse());
DECORATE.beforeFinish(span);
span.finish();
final Object contextAttr = request.getAttribute(DD_CONTEXT_ATTRIBUTE);
if (contextAttr instanceof Context) {
request.removeAttribute(DD_CONTEXT_ATTRIBUTE);

final Context context = (Context) contextAttr;
final AgentSpan span = spanFromContext(context);
if (span != null) {
DECORATE.onResponse(span, request.getResponse());
DECORATE.beforeFinish(context);
span.finish();
}
}
}
}
Loading