Missing traces in Tomcat access logs with spring boot and embedded Tomcat server

[jasaltvik]

Hi!

We are running spring boot apps with embedded Tomcat server in Kubernetes (OpenShift) and use the Red Hat OpenTelemetry Kubernetes Operator with auto-instumentation (Red Hat OpenShift distributed tracing data collection, version 0.63.1-4), typically with minimal Instrumentation custom resources (CRs) like this

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: application-instrumentation
spec:
  exporter:
    endpoint: <redacted>
  java:
    env:
    - name: OTEL_METRICS_EXPORTER
      value: none

and k8s annotations like

instrumentation.opentelemetry.io/container-names: "container1,container2"
instrumentation.opentelemetry.io/inject-java: "true"

We can see the traces in the normal application logs (logback), but they are missing from the Tomcat access logs (using https://logback.qos.ch/access.html to get Tomcat access logs in JSON format). I should also mention that logback-access is used through a custom lib that also checks for the traceparent header to inject the traces into the access logs, but the traceparent header is never found. It seems the access logs are written before injection happens? I was also looking into https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md#application-servers and OTEL_INSTRUMENTATION_TOMCAT_ENABLED, but from what I understand this should be enabled by default (and I am not sure this is related anyways).

Any input on what could be the cause of this would be highly appreciated. Is this way of doing it even supported? We're hoping that we've missed something obvious, and that we don't have to heavily redesign this to get it to work.

PS! I will try to get any additional information necessary if needed.

[laurit]

If you are reading the traceparent header from incoming request it will be present only when it was sent with the request. If your request didn't have a traceparent header and you wish to get the current trace id you could use Span.current().getSpanContext().getTraceId().

[mateuszrzeszutek]

The custom traceparent reading code aside, our current instrumentation does not handle logback-access code. We only instrument Logger and the LoggingEvent; while the LogbackValve just goes around and calls the appenders directly (and uses completely different event type).

[jasaltvik]

@laurit / @mateuszrzeszutek : Sorry for the late reply! Thanks for the input. We'll see how we move forward with this.