How to ship trace data to a secured collector endpoint ?

[santoshkashyap]

Hi

I have node js application with following dependencies:

"@opentelemetry/api": "^1.1.0",
"@opentelemetry/exporter-trace-otlp-http": "^0.31.0",
"@opentelemetry/instrumentation-http": "^0.31.0",
"@opentelemetry/resources": "^1.5.0",
"@opentelemetry/sdk-node": "^0.31.0",
"@opentelemetry/sdk-trace-node": "^1.5.0",
"@opentelemetry/semantic-conventions": "^1.5.0"

My tracing.js which uses Authorization header to send credentials to the secure otlp endpoint

const openTelemetry = require('@opentelemetry/sdk-node')
const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http')
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http')
const { diag, DiagConsoleLogger, DiagLogLevel } = require('@opentelemetry/api')

const tracer = require('./utils/loggingUtil').getTracer(__filename)

diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.DEBUG)

const sdk = new openTelemetry.NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: 'https://<otel endpoint>',
    headers: {
      Authorization: 'Basic <base64 encoded username: pwd >'
    }
  }),
  instrumentations: [new HttpInstrumentation()],
  serviceName: 'my-service'
 
})

sdk.start().then(tracer.debug('Completed OTel Auto-instrumentation for the service.'))

When I run the application :node --require ./src/tracing.js ./src/service.js, I receive below error:
{"stack":"OTLPExporterError: Not Found\n at IncomingMessage.<anonymous> app-router/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:49:55\n at AsyncLocalStorage.run (async_hooks.js:303:14)\n at AsyncLocalStorageContextManager.with (/Users/app-router/node_modules/@opentelemetry/context-async-hooks/build/src/AsyncLocalStorageContextManager.js:33:40)\n at IncomingMessage.contextWrapper (/Users/app-router/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:49:32)\n at IncomingMessage.emit (events.js:412:35)\n at endReadableNT (internal/streams/readable.js:1334:12)\n at processTicksAndRejections (internal/process/task_queues.js:82:21)","message":"Not Found","name":"OTLPExporterError","data":"Status: 404\nDescription: Not Found\n","code":"404"}

I have also tried an alternative approach of specifying the credentials via env variables while running the application. It doesn't work either.
OTEL_EXPORTER_OTLP_ENDPOINT=https://<otel-endpoint> and
OTEL_EXPORTER_OTLP_TRACES_HEADERS='Authorization=Basic base64 encoded(username:pwd)'

I'm not sure what is missing in my configuration. Any input appreciated.

Regards,
Santosh

P.S: The setup works fine when shipping traces to a in-secure http endpoint (eg: jeager)

[santoshkashyap]

UDPATE: When I used env variables, I've also adjusted the NodeSDK creation:

Env Variables
OTEL_EXPORTER_OTLP_ENDPOINT=https://<otel-endpoint> who and
OTEL_EXPORTER_OTLP_TRACES_HEADERS='Authorization=Basic base64 encoded(username:pwd)'

NodeSdk
const sdk = new openTelemetry.NodeSDK({
traceExporter: new OTLPTraceExporter(), // url and headers passed via env variables while running the app
instrumentations: [new HttpInstrumentation()],
serviceName: 'my-service'
})

Error in this case:

{"stack":"OTLPExporterError: Unauthorized\n   
 at IncomingMessage.<anonymous> (/Users/code/cpaxref/app-router/node_modules/@opentelemetry/otlp-exporter-base/build/src/platform/node/util.js:76:35)\n   
 at /Users/code/cpaxref/app-router/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:49:55\n    
at AsyncLocalStorage.run (async_hooks.js:303:14)\n  
  at AsyncLocalStorageContextManager.with (/Users/i324363/code/cpaxref/app-router/node_modules/@opentelemetry/context-async-hooks/build/src/AsyncLocalStorageContextManager.js:33:40)\n  
  at IncomingMessage.contextWrapper (/Users/i324363/code/cpaxref/app-router/node_modules/@opentelemetry/context-async-hooks/build/src/AbstractAsyncHooksContextManager.js:49:32)\n  
  at IncomingMessage.emit (events.js:412:35)\n  
  at endReadableNT (internal/streams/readable.js:1334:12)\n  
  at processTicksAndRejections (internal/process/task_queues.js:82:21)","message":"Unauthorized","name":"OTLPExporterError","data":"<html>\r\n<head><title>401 Authorization Required</title></head>\r\n<body>\r\n<center><h1>401 Authorization Required</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n","code":"401"}
@opentelemetry/instrumentation-http outgoingRequest on request close()

[AccessoSanif]

does that worked for you. I'm in similar spot

[Utwo]

There is a bug in the way opentelemetry-sdk parses the environment variable. You should not have any = sign in the encoded base64 username:pass

[finkinfridom]

We have the same issue with an external service that uses https and needs authentication (so it doesn't seem related to traces only).
any other suggestions?
How could we avoid this? at the moment I'm completely blocked.

[Pratap22]

I'm also getting the same issue.

{"stack":"OTLPExporterError: Not Found\n    at IncomingMessage.<anonymous> (/Users/admin/personal/otel/otel-collector-sender/node_modules/@opentelemetry/otlp-exporter-base/build/src/platform/node/util.js:103:39)\n    at IncomingMessage.emit (node:events:402:35)\n    at endReadableNT (node:internal/streams/readable:1343:12)\n    at processTicksAndRejections (node:internal/process/task_queues:83:21)","message":"Not Found","name":"OTLPExporterError","data":"404 page not found\n","code":"404"}

#2675 (comment)

[zszepczynski]

Is there a solution to it?