OpenTelemetry Debug Demos (KubeCon Japan 2025)

This repository contains language-specific demo applications designed to simulate common OpenTelemetry misconfigurations — and walk through how to debug them. Each demo starts in a broken state and guides you to identify and fix the issue using OpenTelemetry diagnostics, SDK logs, and a local Collector with a debug exporter.

These demos were created for the KubeCon+CloudNativeCon Japan 2025 talk: “Debugging OpenTelemetry: From Missing Signals to Confident Insights.”

Languages covered:

• ☕ Java (missing agent)
• 🍜 Node.js (misconfigured OTLP exporter)
• 🌸 Go (missing tracer provider)

Each service has a light Japanese theme to match the setting in Tokyo 🇯🇵

Prerequisuites

• Docker

Local Debug OpenTelemetry Debug Collector

To debug OpenTelemetry instrumentation, it's often helpful to run a local OpenTelemetry Collector that doesn't forward data to any backend — it just prints incoming telemetry to stdout. This lets you see exactly what your app is sending, without setting up a full observability stack.

You can do that with a single Docker command:

docker run -p 4317:4317 -p 4318:4318 --rm otel/opentelemetry-collector --config=/etc/otelcol/config.yaml --config="yaml:exporters::debug::verbosity: detailed"

🔍 What this command does:

• -p 4317:4317: Exposes the OTLP/gRPC port (default for most SDKs).
• -p 4318:4318: Exposes the OTLP/HTTP port (used in Node.js and some SDKs).
• --rm: Automatically removes the container after it exits.
• otel/opentelemetry-collector: The official Collector Docker image.
• --config=/etc/otelcol/config.yaml: Base config (empty placeholder, required by entrypoint).
• --config="yaml:exporters::debug::verbosity: detailed": Injects an in-memory config that:
  • Enables the debug exporter.
  • Sets it to detailed mode, which prints every span/metric/log it receives to the console.

This is perfect for local debugging — if your app is emitting telemetry, you’ll see it immediately in your terminal. If you don’t, something’s broken, and this Collector helps you isolate where.

For ease, I've wrapped this in the script debug-collector.sh

./debug-collector.sh

Make sure to run the local collector before running the following demos.

If you prefer a more visual approach, I've setup an OpenTelemetry Collector which forwards metrics to Prometheus, traces to Jaeger, and logs to Opensearch.

You can run this using docker compose:

docker compose up

This will bring up the following:

• OpenTelemetry Collector receiving OTLP on 4317 and 4318
• Metrics will be available in Prometheus on http://localhost:9090
• Traces will be available in Jaeger on http://localhost:16686
• Logs will be available in OpenSearch on http://localhost:5601

Environment Variable Configuration

All demos support standard OpenTelemetry environment variables for easy configuration:

Common Variables:

• OTEL_SERVICE_NAME - Sets the service name (defaults: tea-service, ramen-service, sakura-service)
• OTEL_SERVICE_VERSION - Sets the service version
• OTEL_EXPORTER_OTLP_ENDPOINT - OTLP collector endpoint (default: http://localhost:4317 for gRPC)
• OTEL_EXPORTER_OTLP_PROTOCOL - Protocol to use (grpc or http, varies by language)
• OTEL_EXPORTER_OTLP_INSECURE - Allow insecure connections (default: false)
• OTEL_RESOURCE_ATTRIBUTES - Additional resource attributes (e.g., deployment.environment=demo)

Debug Variables:

• OTEL_LOG_LEVEL=debug - Enable verbose SDK logging
• OTEL_TRACES_EXPORTER=stdout/console - Print traces to terminal
• OTEL_METRICS_EXPORTER=stdout/console - Print metrics to terminal
• OTEL_LOGS_EXPORTER=stdout/console - Print logs to terminal

Language-Specific:

• Java: OTEL_JAVAAGENT_DEBUG=true, JAVA_TOOL_OPTIONS="-javaagent:..."
• Node.js: Uses console exporter (not stdout)
• Go: Uses stdout exporter

Java Demo: "Tea Ceremony Tracer" ☕

A small Spring Boot web application that serves a /tea endpoint returning a random Japanese tea type and temperature. It starts without the OpenTelemetry Java Agent attached, so no telemetry is emitted until the agent is properly configured. This demo highlights the importance of runtime instrumentation and shows how missing agents result in silent observability gaps.

Run the broken application:

./run-demo-broken.sh

What is broken?
Forget to attach the javaagent jar.

Symptoms:
App runs, but no trace output. No logs from the agent.

When the Java agent is correctly attached it will output a few log lines before the application starts:

OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[otel.javaagent 2025-05-19 10:35:53:251 +0200] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 2.5.0

Fixing the issue
Adding download and attach the java agent:

curl -L -o opentelemetry-javaagent.jar https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.5.0/opentelemetry-javaagent.jar
java -javaagent:./opentelemetry-javaagent.jar -jar target/tea-service-0.1.0.jar

or using the JAVA_TOOL_OPTIONS environment variable

export JAVA_TOOL_OPTIONS="-javaagent:./opentelemetry-javaagent.jar"

Now run the demo again with these fixes:

./run-demo-fixed.sh

Additional tips for debugging The run-demo-debug.sh configures debug logging for both the Java Agent and the OpenTelemetru SDK.

OTEL_JAVAAGENT_DEBUG=true

Controls debug output from the Java agent itself, and allows for inspection of e.g. which libaries will be auto-instrumented.

Whereas:

OTEL_LOG_LEVEL=debug

controls logging inside the OpenTelemetry SDK and exporters, and e.g. allows you to see resource attributes included in traces.

Node.js Demo – Ramen Ratings API 🍜

A lightweight Express application that serves a /ramen endpoint returning a random ramen type and a rating between 0–5. The app uses OpenTelemetry’s auto-instrumentation for Node.js by requiring it at startup via the NODE_OPTIONS environment variable.

This demo begins with a common misconfiguration: Node.js defaults to the OTLP protocol http/protobuf, but the local OpenTelemetry Collector is listening on port 4317, which expects grpc. The result is that telemetry is silently dropped.

Run the broken application:

./run-demo-broken.sh

What is broken? The Node.js SDK defaults to http/protobuf, but the OTLP endpoint (localhost:4317) expects grpc.

Symptoms: The application starts normally, and /ramen returns JSON data, but protocol mismatch errors appear:

Error: Parse Error: Expected HTTP/
code: "HPE_INVALID_CONSTANT"
reason: "Expected HTTP/"

This occurs because the HTTP client is trying to parse a gRPC response from port 4317.

Fixing the issue Configure the correct protocol explicitly to match the port:

export OTEL_EXPORTER_OTLP_PROTOCOL=grpc

Then run the fixed version:

./run-demo-fixed.sh

Additional tips for debugging

Run the debug version to verify telemetry generation using console exporters:

./run-demo-debug.sh

This script uses console exporters instead of sending to the collector:

export OTEL_TRACES_EXPORTER=console
export OTEL_METRICS_EXPORTER=console  
export OTEL_LOGS_EXPORTER=console

What it shows:

• Detailed debug logs from the OpenTelemetry SDK
• JSON-formatted traces, metrics, and logs directly in your terminal
• Confirms that telemetry is being generated locally without needing a collector
• Shows automatic Express.js and HTTP instrumentation in action

Example output:

{
  "name": "GET /ramen",
  "kind": 1,
  "attributes": {
    "http.method": "GET",
    "http.route": "/ramen"
  }
}

Go Demo – Sakura Stats Service 🌸

A minimal HTTP service in Go that exposes a /sakura endpoint returning fictional cherry blossom bloom data. The application includes full observability with traces, metrics, and structured logs that include trace correlation data.

The demo starts in a broken state with no TracerProvider configured — a common pitfall in Go applications where telemetry appears to be integrated, but all signals are silently dropped.

This demo highlights how using OpenTelemetry's API without setting up the SDK results in no telemetry being emitted, even though the application compiles and runs correctly.

Run the broken version

./run-demo-broken.sh

This runs broken/main.go, which uses the OpenTelemetry API to start spans — but never sets a tracer provider or exporter.

Symptoms:

• The application runs normally at http://localhost:8080/sakura
• No telemetry is exported — spans are silently dropped
• No warnings or errors are shown, making this a subtle issue

Run the fixed version

./run-demo-fixed.sh

This runs fixed/main.go, which includes:

• A gRPC-based OTLP exporter targeting localhost:4317
• A properly initialized TracerProvider and resource configuration
• Clean shutdown with span flushing on exit

Expected outcome:

• Traces for the GetSakuraStats operation appear in your collector
• Metrics showing request counts and petal count distributions
• Structured logs with trace correlation (trace_id, span_id) in OpenSearch

Additional tips for debugging

./run-demo-debug.sh

This runs the fixed app but:

• Sets OTEL_TRACES_EXPORTER=stdout
• Sets OTEL_LOGS_EXPORTER=stdout
• Sets OTEL_METRICS_EXPORTER=stdout
• Enables detailed SDK logs with OTEL_LOG_LEVEL=debug

This mode prints telemetry directly to your terminal without needing a collector, helping you verify that spans are created correctly.

Common Pitfall in Go Using the OpenTelemetry API without setting a TracerProvider results in a noop tracer. Spans are silently discarded unless the SDK is configured:

otel.SetTracerProvider(sdktrace.NewTracerProvider(...))

This demo makes that failure mode visible.

Enhanced Features

Trace Correlation

All services now include trace correlation in their logs:

• Java: Automatic via OpenTelemetry Logback appender
• Node.js: Via Winston OpenTelemetry transport
• Go: Manual trace_id/span_id injection in structured logs

Full Observability Stack

Each demo generates:

• ✅ Traces with automatic HTTP instrumentation
• ✅ Metrics (request counts, response times, business metrics)
• ✅ Logs with structured data and trace correlation

Multiple Export Options

• Production: OTLP to collector → Jaeger/Prometheus/OpenSearch
• Debug: Console/stdout exporters for local verification
• Hybrid: Environment variable switching between modes