datadog_agent Traces forwarding

[tbenade]

Hi, we observe several Kubernetes clusters using the Datadog agent (traces & metrics) and Vector Agent for log forwarding to a Vector Aggregation service and on to Datadog.

In our DatadogAgent -> Vector Aggregator -> Datadog for traces we are seeing no data in the Vector Aggregation Service (yip have multiple_outputs: true set). The bewildering bit is even with the Datadog agent configured for alternate data destination the data still arrives at Datadog but never our Vector Aggregation service. Metrics are working perfectly fine.

We use the Datadog operator to deploy the Datadog agents with the following envars configured.

    DD_OBSERVABILITY_PIPELINES_WORKER_TRACES_ENABLED: "true"
    DD_OBSERVABILITY_PIPELINES_WORKER_TRACES_URL: "https://amazing.aggregator.com:8282"
    DD_OBSERVABILITY_PIPELINES_WORKER_METRICS_ENABLED: "true"
    DD_OBSERVABILITY_PIPELINES_WORKER_METRICS_URL: "https://amazing.aggregator.com:8282"
    DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED: "true"
    DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL: "https://amazing.aggregator.com:8282"

https://github.com/DataDog/datadog-agent/blob/main/pkg/config/config_template.yaml#L676

We are totally stumped. Appreciate this is a blend of Datadog and Vector questions, but we are all the same family here 😄 so hoping the the gurus have an insight as to what might be happening. Thank you.

[pront]

Hi @tbenade, a few of questions to help narrow this down.

1. Can you share your configuration? (reminder to redact sensitive data if any)
2. Was this working before? Any changes?
3. What's your Vector version
4. What's your DD agent version?

[tbenade]

1. Can you share your configuration? (reminder to redact sensitive data if any)

# vector - datadog_agent
type: datadog_agent
address: 0.0.0.0:8282
store_api_key: true
multiple_outputs: true

#datadog-agent template in datadog-operator deployment
# really not sure what you want from the dd-agent so included small excerpt
containers:
      agent:
        env:
          - name: DD_CHECKS_TAG_CARDINALITY
            value: "orchestrator"
          - name: DD_OBSERVABILITY_PIPELINES_WORKER_METRICS_ENABLED
            value: {{ .Values.datadogAgent.env.DD_OBSERVABILITY_PIPELINES_WORKER_METRICS_ENABLED | quote }}
          - name: DD_OBSERVABILITY_PIPELINES_WORKER_METRICS_URL
            value: {{ .Values.datadogAgent.env.DD_OBSERVABILITY_PIPELINES_WORKER_METRICS_URL | quote }}
          - name: DD_OBSERVABILITY_PIPELINES_WORKER_TRACES_ENABLED
            value: {{ .Values.datadogAgent.env.DD_OBSERVABILITY_PIPELINES_WORKER_TRACES_ENABLED | quote }}
          - name: DD_OBSERVABILITY_PIPELINES_WORKER_TRACES_URL
            value: {{ .Values.datadogAgent.env.DD_OBSERVABILITY_PIPELINES_WORKER_TRACES_URL | quote }}
          - name: DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_ENABLED
            value: {{ .Values.datadogAgent.env.DD_OBSERVABILITY_PIPELINES_WORKER_TRACES_ENABLED | quote }}
          - name: DD_OBSERVABILITY_PIPELINES_WORKER_LOGS_URL
            value: {{ .Values.datadogAgent.env.DD_OBSERVABILITY_PIPELINES_WORKER_METRICS_URL | quote }}
      processAgent:
        env:
          - name: DD_ORCHESTRATOR_EXPLORER_ENABLED
            value: "true"

2. Was this working before? Any changes?

Nope a new deployment

3. What's your Vector version

0.45.0

4. What's your DD agent version?

gcr.io/datadoghq/agent:7.63.3

[tbenade]

Hi @pront, just checking in if you had any more feedback or if this thread is dead. Thanks 🙏

[tbenade]

HI, thanks for coming back to me. We are seeing zero trace events received by datadog_agent source. And the other events balance given we have multiple_outputs: true. datadog_logs can be ignored that is being sent by a Vector agent directly. Metrics aside it is more concerning that the traces are still reaching Datadog API even after we have configured.

 traces:
     enabled: true
     url: "http//127.0.0.1:8080"

Our current belief is that the Datadog agent does not actually support traces through observability_pipelines_worker configuration and ignores the setting entirely.

[pront]

Hi @tbenade, it sounds like traces are never hitting your Vector instance but go directly to your Datadog org? That might indicate a DD agent configuration issue. I will try to setup a basic example on my side with the config snippets I provided above.

[pront]

I tried everything together on my laptop and it works as expected.

A few notes:

• observability_pipelines_worker and vector can be used interchangeably, let's use vector from now on to avoid confusion
• the DD agent provides a GUI at http://localhost:5052/, this is very neat because it show any errors that weren't caught at startup e.g. bad URL. The http//127.0.0.1:8080 is missing a :, it should be http://127.0.0.1:8080
• if you want to exclusively forward to vector, you need to disable additional endpoints in your DD agent config

My vector config:

sources:
  dd_agent:
    type: datadog_agent
    address: 127.0.0.1:8137
    multiple_outputs: true

sinks:
  logs_sink:
    type: datadog_logs
    inputs:
      - dd_agent.logs
    default_api_key: X

  metrics_sink:
    type: datadog_metrics
    inputs:
      - dd_agent.metrics
    default_api_key: X

  traces_sink:
    type: datadog_traces
    inputs:
      - dd_agent.traces
    default_api_key: X

  console:
    type: console
    inputs:
      - dd_agent.traces
    encoding:
      codec: json
      json:
        pretty: true

[pront]

For posterity and if it helps anyone in the future, I recorded everything here: https://github.com/vectordotdev/issue-examples/tree/main/discussions/22828

[tbenade]

@pront firstly I can't thank you enough for having a go, truly appreciated. Secondly this is really good news and we will keep poking that DD agent config on our end given your evidence above. Much appreciated 🙏

[tbenade]

Many many hours later still cannot get this to work. Given we put exactly the same value in the traces URL as metrics URL which works perfectly this cannot be a configuration issue.

I didn't understand this comment "you need to disable additional endpoints in your DD agent config". Given we don't do that currently and metrics work, Im still convinced forwarding of traces from DDog agent to Vector does not work.

[pront]

I didn't understand this comment "you need to disable additional endpoints in your DD agent config".

The DD agent can be configured to dual ship to vector and other endpoints. If you inspect the status pages you should see it under APM.

When your setup works you should see receiver (DD agent ingestion) and writer (towards Vector) traffic:

The error logs:

[pront]

Also, I hope your url in the DD agent is correct, for example:

vector:
   metrics:
     enabled: true
     url: http://127.0.0.1:8137

And you fully restart the agent each time its config changes. On my local setup I did it with: launchctl kickstart -k gui/501/com.datadoghq.agent.

Otherwise, it might be a networking issue on your environment.

[tbenade]

Hi @pront thanks again for trying to help. I went away an built the simplest possible Docker solution of App,DD agent and Vector agent and can confirm I was finally able to send the traces from DD Agent to Vector and then on to Datadog no problem. Unfortunately I still have no idea why the same plan is not working in our K8 cluster when using the Datadog operator too deploy our agents. No matter how we configure forwarding the Datadog agent always forwards the traces to Datadog while it will happily forward metrics and logs to Vector. Bizarre till I find my typo or something else silly 😄