fix(dnstap source): implement all TCP dnstap options to reduce error

[esensar]

Summary

The dnstap TCP source was initially built based on the TCP socket source, but not all of the options were correctly implemented, which left the request limiter unused. This also changes the multithreaded approach to do async-aware waits, because previous implementation used thread sleep in async context.

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

How did you test this PR?

Besides the included tests, this was also tested in real environments and it has not yet had errors like described in #20744.

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the "no-changelog" label to this PR.

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• The CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • cargo fmt --all
    • cargo clippy --workspace --all-targets -- -D warnings
    • cargo nextest run --workspace (alternatively, you can run cargo test --all)
    • ./scripts/check_changelog_fragments.sh
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run cargo vdev build licenses to regenerate the license inventory and commit the changes (if any). More details here.

References

• Related: DNSTAP socket shows errors after operating for some time #20744

---

Sponsored by Quad9

[johnhtodd]

We've been running this patch in production on half a dozen large collectors (probably half a trillion records processed?) for several days, with no issues. This has fixed our problems with dnstap sockets falling over and not recovering. Very much in favor of inclusion.

[pront]

Let's extract this into a constant and also document why this value was selected.

[esensar]

I copied this over from src/sources/util/tcp/mod and I can see it was added in b8fb1e3 . Not really sure why this value was selected, I guess the goal was to pick a fairly short time, but still enough for the connection to be able to take a request.

[pront]

I see, it's an existing code smell. From the PR:

A small read timeout was added, and if a connection does not receive any data during that time it will release its permit (and try to obtain a new one) allowing other connections to read.

We can use this as a comment here.

[pront]

Thanks!