Configure DataDog agent to send HTTP metric data to Vector

[fenill1]

Hello,

I have a Datadog agent to send HTTP metric data to vector, however I am not able to see any HTTP metric data if reading through the <component_id>.metrics as suggested in the Vector documentation. configuring-the-datadog-agent

This is the datadog configuration (yaml)

vector:
 metrics.enabled: true
 metrics.url: http://0.0.0.0:6901

And this is the vector configuration (yaml)

sources:
  sample_agent:
    type: "datadog_agent"
    multiple_outputs: true
    address: "0.0.0.0:6901"

sinks:
  sample_metrics:
    type: "console"
    inputs:
      - "sample_agent.metrics"
    encoding:
      codec: "json"

The only way I am able to see any HTTP metric data is if reading through <component_id>.traces.

This is the datadog configuration (yaml)

vector:
 traces.enabled: true
 traces.url: http://0.0.0.0:6901

And this is the vector configuration (yaml)

sources:
  sample_agent:
    type: "datadog_agent"
    multiple_outputs: true
    address: "0.0.0.0:6901"

sinks:
  sample_traces:
    type: "console"
    inputs:
      - "sample_agent.traces"
    encoding:
      codec: "json"

But for this to show 100% accurate data we have to set the -Ddd.trace.sample.rate=1 which is not sustainable.

The question I have and would like assistance on is to understand if there is a better way to read through the HTTP metric data instead of doing it through traces. Is there anything else I need to do in order to get Datadog to output HTTP metric data to vector?

Thanks in advance for any help!

[pront]

Hi @fenill1,

You have set multiple_outputs to true, which is desirable here. This means the sample_agent source will have three outputs, sample_agent.logs, sample_agent.metrics and sample_agent.traces.

All your configurations look good.

The question I have and would like assistance on is to understand if there is a better way to read through the HTTP metric data instead of doing it through traces. Is there anything else I need to do in order to get Datadog to output HTTP metric data to vector?

You should only use sample_agent.metrics for metrics. If you are not seeing any traffic, a number of things might be going on.

1. What is your Vector version?
2. What is your DD agent version?
3. Any interesting debug or error logs in Vector or in the agent?

Note: There is a caveat when it comes to traces, see here.

[fenill1]

Hello @pront

Thank you for reviewing this.

Vector version is 0.44.0
DD agent version is 1.46.1
There are no errors we are seeing in any vector logs.

Note: I have set the -Ddd.trace.sample.rate=1 which should send 100% of the request data per my understanding. Reducing this value impacts amount the traces we see being sent through.

Does this mean in order to see HTTP metric data we can do that only through the sample_agent.traces with the sample rate set to 1 in the DataDog Agent and not through sample_agent.metrics?

Thank you.

[pront]

Note: I have set the -Ddd.trace.sample.rate=1 which should send 100% of the request data per my understanding. Reducing this value impacts amount the traces we see being sent through.

This is correct.

Does this mean in order to see HTTP metric data we can do that only through the sample_agent.traces with the sample rate set to 1 in the DataDog Agent and not through sample_agent.metrics?

Yes. The sample_agent.traces output will exclusively output traces.

[fenill1]

I understand that to get traces data we have to use sample_agent.traces but how about getting HTTP metric data?

[pront]

All metrics should be part of the sample_agent.metrics stream. If I understand correctly, you aren't seeing any events coming out of that source port?

You can easily validate this with vector top or vector tap, see https://vector.dev/guides/developer/debugging/#vector-tools