Shutdown behaviour of Java Agent

[otbe]

Hello,

recently I was facing a tricky debugging situation. In one of our services a situation happened (when looking at the state of the system) that should result in multiple log lines (error level with exceptions). It happened during the shutdown (deployment) of this service and it seems like we regularly miss logs from a graceful shutdown.

What we use:

• spring boot 3.5.x
• latest Otel java agent
• logback with logback instrumentation via agent (so we only have a NOPAppender configured in our logback.xml)
• we deploy on AWS Fargate

There are certain logs from Spring itself (when the embedded tomcat shuts down) but we also have custom logs via Slf4J in destroy() methods that are managed by Spring. Sometimes we see these logs but not always. Sometimes all logs are shipped, sometimes only a few of them and sometimes nothing.

For debugging purposes I now added a ConsoleAppender to our logback configuration and it seems to work. All logs are always visible.

I wonder what the shutdown behaviour of the java agent is at this point? Was searching in the docs but couldn't find anything.
Is the Java agent also somehow closing at some point and won't ship any logs after this point?

Thanks for your input :)

[trask]

The Java agent registers a JVM shutdown hook to flush any buffered telemetry: https://github.com/open-telemetry/opentelemetry-java/blob/a67d37bf7025095e52fccc8fcf6ed0de8510d801/sdk-extensions/autoconfigure/src/main/java/io/opentelemetry/sdk/autoconfigure/AutoConfiguredOpenTelemetrySdkBuilder.java#L605

Are these logs you're expecting also emitted from (another) JVM shutdown hook?

[otbe]

Yes they're emitted during the closing of the spring context (via DisposableBean, @PreDestroy...) which is triggered from one shutdown hook Spring registers by itself. To be honest, Im not quite sure how it can work but it was still surprising for me 😄