fix(datadog_agent source): Silence the "Source send cancelled." error

[bruceg]

Summary

Eliminated the "Source send cancelled." error and corresponding metric for the datadog_agent source, as Datadog Agent will always resend events when the connection is dropped after a timeout.

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

How did you test this PR?

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the "no-changelog" label to this PR.

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• The CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • cargo fmt --all
    • cargo clippy --workspace --all-targets -- -D warnings
    • cargo nextest run --workspace (alternatively, you can run cargo test --all)
    • ./scripts/check_changelog_fragments.sh
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run cargo vdev build licenses to regenerate the license inventory and commit the changes (if any). More details here.

[Copilot]

Pull Request Overview

This PR silences the "Source send cancelled." error and its corresponding metric for the datadog_agent source by disabling the unsent event counting behavior.

• Updated datadog_agent source to call silence_unsent_events() on the SourceSender.
• Refactored UnsentEventCount in SourceSender to optionally count events using an Option.
• Added a changelog fragment to document the change.

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
src/sources/datadog_agent/mod.rs	Modified the warp filter builder to mute the unsent event count error by silencing events.
src/source_sender/mod.rs	Added silence_unsent_events() and refactored UnsentEventCount to optionally disable counting.
changelog.d/fix-datadog-agent-source-sender-warning.fix.md	Provided a changelog entry summarizing the fix.

[datadog-vectordotdev]

Datadog Report

Branch report: bruceg/OPA-3143-datadog-agent
Commit report: 216ae08
Test service: vector

❌ 1 Failed (1 Known Flaky), 7 Passed, 0 Skipped, 25.47s Total Time

❌ Failed Tests (1)

• datadog::logs::validate - vector::e2e - ❄️ Known flaky - Details

  Expand for error

   Test has failed
  

[bruceg]

This is reliably breaking the datadog-logs E2E test. It appears that the agent-vector configuration is not delivering logs to Vector, but I can't see why. I have reproduced it on my system, though trying the same test current master also fails, so I have no idea where to go on this.

[byronwolfman]

Hello,

I'm not sure if you're taking feedback on this PR, but my team uses Vector extensively and we find this error and associated metric very useful. "Source send cancelled" due to a timeout can mean that Vector is saturated and/or that a component is applying backpressure. "Source send cancelled" can be a symptom of various capacity issues including (but not limited to) saturated sink buffers, undersized compute, or even just general slowness on whatever's on the other side of the sink (e2e acknowledgements affect some but not all of these).

If possible, we'd like to keep these metrics and log events. They act as an early warning sign for us because even though the Datadog Agent will retransmit, an increase in "Source send cancelled" errors means that we are beginning to erode our safety margin.