Full OpenTelemetry Support for Lambda Extension

[naftulikay]

I filed recently #715 and @duncanista advised me to create an issue here to track a feature request.

Background

I've been working on a project for quite some time which will run as an AWS Lambda function serving HTTP requests via API Gateway, asynchronous queue jobs via SQS, scheduled jobs (i.e. cron) with EventBridge Scheduler, and a handful of other services. My application is written in Rust, so since there is no native Datadog crate yet, I was planning on using Datadog's OpenTelemetry support to send my logs, metrics, and traces to Datadog. I've done this before with a non-Lambda Rust application in Kubernetes and it works great.

I'm aware that I can configure Datadog in AWS to pull logs from CloudWatch, but as far as I can tell, there is no Rust crate available that can emit logs in OpenTelemetry JSON format, and I'm not sure if there's even a standard for this whatsoever. opentelemetry_stdout emits debug-formatted telemetry which is neither really machine-parseable nor human-readable.

@duncanista advised me that this newer Lambda extension supports OTEL traces but does not yet support logging and metrics, so I'm adding this issue as a feature request.

Feature Request

Adding support for the full suite of OpenTelemetry data (logs, traces, and metrics) would enable me (and others) to send all telemetry to Datadog through the OpenTelemetry standard, regardless of which programming language is being used.

As for the actual wire format, I'm agnostic as to whether to send telemetry over HTTP or gRPC, Rust can do either just fine.

In summary, I'm asking for support for this extension to be added for full OpenTelemetry support in terms of logging, tracing, and metrics, and I'm hoping this won't be too difficult to add, as the Datadog Agent itself supports everything I need.

[duncanista]

Hey @naftulikay,

While we work on supporting OTEL metrics and logs, I wanted to ask you a couple questions:

I'm aware that I can configure Datadog in AWS to pull logs from CloudWatch, but as far as I can tell, there is no Rust crate available that can emit logs in OpenTelemetry JSON format, and I'm not sure if there's even a standard for this whatsoever. opentelemetry_stdout emits debug-formatted telemetry which is neither really machine-parseable nor human-readable.

Our Lambda Extension currently receives all logs from STDOUT and emits them to Datadog by default, you don't have to configure anything else for this to work.

Do you still think you need something to get Logs from here?

On metrics, what type of metrics are you looking to send? Currently, and by default, you can send metrics to localhost:8125 and our StatsD server will pick them up automatically and forward them to Datadog OOTB.

I just wanted to point this out in case you were not aware of them, maybe something we can work from our side in our docs.

–––

Thanks for raising the issue! While we work on it, I wanted to make sure you had a workaround in the meantime.

If there are any questions, let me know! (we are still working on adding docs for rust and improve our OTLP config for Lambda, we haven't forgotten about it!)

Also, keep an eye on our page, we might be releasing a Rust tracer soon :) 🦀

[naftulikay]

@duncanista yes, I'm aware that logs will be collected from stdout and I'm aware that I can emit statsd, but I'm hoping to get OpenTelemetry support so that all of the telemetry has correlations enabled. When I've hosted Rust applications in Kubernetes with the Datadog Agent installed, it's awesome to see tracing spans and view logs that exclusively occurred during those spans.

I'll keep my eye out for the Rust tracer :)