SigV4 Authentication Support for OTLP HTTP Logs Exporter #181

liustve · 2025-05-12T20:17:10Z

Issue #, if available:
Supporting ADOT JS auto instrumentation to automatically inject SigV4 authentication headers for outgoing log requests to the allow exporting to the AWS Logs OTLP endpoint. Users will need to configure the following environment variables in order to enable and properly run this exporter:

OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://logs.[AWS-REGION].amazonaws.com/v1/logs; required
OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=[CW-LOG-GROUP-NAME],x-aws-log-stream=[CW-LOG-STREAM-NAME] required
OTEL_LOGS_EXPORTER=otlp required or do not set env variable
OTEL_EXPORTER_OTLP_LOGS_PROTOCOL=http/protobuf required or do not set env variable
OTEL_METRICS_EXPORTER=none

This feature currently supports only 2 logging libraries by auto-instrumentation, Bunyan and Winston:
https://docs.honeycomb.io/send-data/logs/opentelemetry/sdk/javascript/

Description of changes:

Add new AwsAuthenticator class used by both OtlpAwsLogExporter and OtlpAwsSpanExporter which extends the upstream OTLPProtoLogExporter to inject Sigv4 headers directly into the headers.
Modified ADOT JS auto instrumentation to automatically detect if a user is exporting to CW Logs OTLP Logs endpoint by checking if the environment variable OTEL_EXPORTER_OTLP_LOGS_ENDPOINT is configured to match this url pattern: https://logs.[AWS-REGION].amazonaws.com/v1/logs

Testing:

E2E test done in an empty EC2 environment without configuring .aws credentials config file or setting AWS credentials in the environment variable
Manual testing was done by configuring the above environment variables and setting up the sample app locally with ADOT auto instrumentation and verified the logs in CW Logs.
Unit tests were added to verify functionality of OtlpAwsLogsExporter

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Example log in CW Logs:

{
    "resource": {
        "attributes": {
            "service.name": "unknown_service:/home/ec2-user/.local/share/mise/installs/node/22.14.0/bin/node",
            "process.command_args": [
                "/home/ec2-user/.local/share/mise/installs/node/22.14.0/bin/node",
                "--experimental-network-inspection",
                "--require",
                "@aws/aws-distro-opentelemetry-node-autoinstrumentation/register",
                "/home/ec2-user/aws-otel-js-instrumentation/sample-applications/simple-express-server/sample-app-express-server.js"
            ],
            "process.runtime.version": "22.14.0",
            "process.pid": 1599620,
            "process.executable.name": "/home/ec2-user/.local/share/mise/installs/node/22.14.0/bin/node",
            "telemetry.sdk.name": "opentelemetry",
            "process.owner": "ec2-user",
            "telemetry.sdk.language": "nodejs",
            "process.runtime.name": "nodejs",
            "process.executable.path": "/home/ec2-user/.local/share/mise/installs/node/22.14.0/bin/node",
            "host.arch": "amd64",
            "telemetry.sdk.version": "1.30.1",
            "process.command": "/home/ec2-user/aws-otel-js-instrumentation/sample-applications/simple-express-server/sample-app-express-server.js",
            "host.name": "ip-172-31-7-29.us-west-2.compute.internal",
            "process.runtime.description": "Node.js",
            "telemetry.auto.version": "0.6.0-dev0-aws",
            "host.id": "ec2ccd3acc52be039f977d9b1de7c64d"
        }
    },
    "scope": {
        "name": "default"
    },
    "timeUnixNano": 1748220324612000000,
    "observedTimeUnixNano": 1748220324612000000,
    "severityNumber": 9,
    "severityText": "INFO",
    "body": "Received request to /rolldice",
    "attributes": {
        "endpoint": "/rolldice"
    },
    "flags": 1,
    "traceId": "6833b9a4cccf4d9bfb82b11686fd8f63",
    "spanId": "bd19f2fcbc107755"
}

jj22ee · 2025-06-12T19:16:38Z

This feature currently supports only 2 logging libraries by auto-instrumentation, Bunyan and Winston:
https://docs.honeycomb.io/send-data/logs/opentelemetry/sdk/javascript/

Can we support Pino as well? We mention it in our docs and is considered top 2 popular logging libraries for Node.js.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Application-Signals-TraceLogCorrelation.html

The Pino, Winston, or Bunyan auto-instrumentations for Node.js.

jj22ee · 2025-06-12T19:42:31Z

aws-distro-opentelemetry-node-autoinstrumentation/src/register.ts

@@ -83,6 +84,9 @@ try {
  diag.info('Setting TraceProvider for instrumentations at the end of initialization');
  for (const instrumentation of instrumentations) {
    instrumentation.setTracerProvider(trace.getTracerProvider());
+    if (instrumentation.setLoggerProvider) {
+      instrumentation.setLoggerProvider(logs.getLoggerProvider());


Wondering if this is required? Since Global Logger Provider is also set here: https://github.com/open-telemetry/opentelemetry-js/blob/v1.30.1/experimental/packages/opentelemetry-sdk-node/src/sdk.ts#L401

Does this feature without this setLoggerProvider code?

Yep confirmed this works without the setLoggerProvider

...tro-opentelemetry-node-autoinstrumentation/src/exporter/otlp/aws/common/aws-authenticator.ts

aws-distro-opentelemetry-node-autoinstrumentation/package.json

jj22ee · 2025-06-12T20:18:34Z

...pentelemetry-node-autoinstrumentation/src/exporter/otlp/aws/traces/otlp-aws-span-exporter.ts

+   * sending it to the endpoint. Otherwise, we will skip signing.
+   */
+  public override async export(items: ReadableSpan[], resultCallback: (result: ExportResult) => void): Promise<void> {
+    const serializedSpans: Uint8Array | undefined = ProtobufTraceSerializer.serializeRequest(items);


Previously, we return right away and do nothing if serializedSpans was undefined, but not anymore in the new change. Is there a reason for this, was it because it was redundant because the super call would also do nothing (I didn't check the super method logic)?

added the immediate return back if serializedSpans is undefined

...pentelemetry-node-autoinstrumentation/src/exporter/otlp/aws/traces/otlp-aws-span-exporter.ts

jj22ee · 2025-06-12T22:39:33Z

...tro-opentelemetry-node-autoinstrumentation/src/exporter/otlp/aws/common/aws-authenticator.ts

+  // Cleans up Sigv4 from headers to avoid accidentally copying them to the new headers
+  private removeSigV4Headers(headers: Record<string, string>) {
+    const newHeaders: Record<string, string> = {};
+    const sigV4Headers = ['x-amz-date', 'authorization', 'x-amz-content-sha256', 'x-amz-security-token'];


You could move this to be a constant outside of this class.

jj22ee · 2025-06-12T22:42:02Z

...tro-opentelemetry-node-autoinstrumentation/src/exporter/otlp/aws/common/aws-authenticator.ts

+        const signer = new SignatureV4({
+          credentials: defaultProvider(),
+          region: this.region,
+          service: this.service,
+          sha256: Sha256,
+        });


This can be a class member, no need to create this object for every authentication.

I believe we need to since defaultProvider() needs to be called in every export to ensure we're passing in the latest credentials.

jj22ee · 2025-06-13T00:21:51Z

aws-distro-opentelemetry-node-autoinstrumentation/src/aws-opentelemetry-configurator.ts

+
+  for (const pair of logsHeaders.split(',')) {
+    if (pair.includes('=')) {
+      const [key, value] = pair.split('=', 2);


fyi, this makes it so that 'a=b=c'.split('=', 2) become ["a", "b"], not ["a", "b=c"]

You may want to consider this approach if this is a concern:

aws-otel-js-instrumentation/aws-distro-opentelemetry-node-autoinstrumentation/src/aws-opentelemetry-configurator.ts

Lines 299 to 303 in 241fa38

for (const arg of args) {

const equalIndex: number = arg.indexOf('=');

if (equalIndex === -1) {

continue;

}

But based on the logic, you don't care about the value, so it may not be a concern.

jj22ee · 2025-06-13T00:59:07Z

aws-distro-opentelemetry-node-autoinstrumentation/src/aws-opentelemetry-configurator.ts

+  if (filteredLogHeadersCount !== 2) {
+    diag.warn(
+      'Improper configuration: Please configure the environment variable OTEL_EXPORTER_OTLP_LOGS_HEADERS ' +
+        'to have values for x-aws-log-group and x-aws-log-stream'


Suggested change

'to have values for x-aws-log-group and x-aws-log-stream'

`to have values for ${AWS_OTLP_LOGS_GROUP_HEADER} and ${AWS_OTLP_LOGS_STREAM_HEADER}`

jj22ee · 2025-06-13T01:02:57Z

aws-distro-opentelemetry-node-autoinstrumentation/src/aws-opentelemetry-configurator.ts

+    }
+  }
+
+  if (filteredLogHeadersCount !== 2) {


Technically, a user can just set x-aws-log-group twice, and that would cause validateLogsHeaders() to be true. To get around this, you can just split the (key === AWS_OTLP_LOGS_GROUP_HEADER || key === AWS_OTLP_LOGS_STREAM_HEADER) check into an if-elseif statement and just validate that both statements are entered.

liustve · 2025-06-20T17:21:48Z

This feature currently supports only 2 logging libraries by auto-instrumentation, Bunyan and Winston:
https://docs.honeycomb.io/send-data/logs/opentelemetry/sdk/javascript/

Can we support Pino as well? We mention it in our docs and is considered top 2 popular logging libraries for Node.js.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Application-Signals-TraceLogCorrelation.html

The Pino, Winston, or Bunyan auto-instrumentations for Node.js.

Yep should also support Pino as well, I missed that in the description.

initial commit

6d12166

liustve requested a review from a team as a code owner May 12, 2025 20:17

liustve marked this pull request as draft May 12, 2025 20:17

liustve self-assigned this May 12, 2025

sigv4 authentication support for otlp logs exporter

6b3ceb1

liustve force-pushed the sigv4_logs branch from d0cc54a to 6b3ceb1 Compare May 27, 2025 05:22

liustve closed this May 27, 2025

liustve changed the title ~~[WIP] SigV4 Authentication Support for OTLP HTTP Logs Exporter~~ SigV4 Authentication Support for OTLP HTTP Logs Exporter May 28, 2025

liustve reopened this May 28, 2025

liustve marked this pull request as ready for review May 28, 2025 00:28

liustve requested review from srprash and jj22ee May 28, 2025 00:28

liustve mentioned this pull request May 28, 2025

SigV4 Authentication Support for OTLP HTTP Logs Exporter #184

Closed

Merge branch 'main' into sigv4_logs

511de17

jj22ee reviewed Jun 13, 2025

View reviewed changes

liustve added 10 commits June 20, 2025 10:21

Merge branch 'main' into sigv4_logs

c577ac8

add preprocessing for compression

8f753e3

linting fix

9df5a67

add further refactoring

87b6f01

add fix for proper header config

3bf832b

delete utils.ts

93807d7

add better debug messages

694980a

linting fix

cbcb0e9

add bunyan dependency

3827f98

fix sigv4 tests

124f98e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

SigV4 Authentication Support for OTLP HTTP Logs Exporter #181

SigV4 Authentication Support for OTLP HTTP Logs Exporter #181

liustve commented May 12, 2025 •

edited

Loading

Uh oh!

jj22ee commented Jun 12, 2025

Uh oh!

jj22ee Jun 12, 2025

Uh oh!

liustve Jun 20, 2025

Uh oh!

Uh oh!

Uh oh!

jj22ee Jun 12, 2025

Uh oh!

liustve Jun 20, 2025

Uh oh!

Uh oh!

jj22ee Jun 12, 2025

Uh oh!

jj22ee Jun 12, 2025

Uh oh!

liustve Jun 20, 2025

Uh oh!

jj22ee Jun 13, 2025

Uh oh!

jj22ee Jun 13, 2025

Uh oh!

jj22ee Jun 13, 2025

Uh oh!

liustve commented Jun 20, 2025

Uh oh!

Uh oh!

	for (const arg of args) {
	const equalIndex: number = arg.indexOf('=');
	if (equalIndex === -1) {
	continue;
	}

	'to have values for x-aws-log-group and x-aws-log-stream'
	`to have values for ${AWS_OTLP_LOGS_GROUP_HEADER} and ${AWS_OTLP_LOGS_STREAM_HEADER}`

SigV4 Authentication Support for OTLP HTTP Logs Exporter #181

Are you sure you want to change the base?

SigV4 Authentication Support for OTLP HTTP Logs Exporter #181

Conversation

liustve commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jj22ee commented Jun 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liustve commented Jun 20, 2025

Uh oh!

Uh oh!

liustve commented May 12, 2025 •

edited

Loading