enhancement(splunk_hec sink): Use a response cookie to route ack checks to the same Splunk indexer

[jvperrin]

Summary

This is particularly useful when running Splunk in a clustered environment with multiple indexer hosts. In this environment, acknowledgement IDs are frequently duplicated across multiple indexers (they all start at 0 and count upwards as they receive requests with the same X-Splunk-Request-Channel header, so there will be lots of reuse). One common way to distinguish between multiple hosts behind a load balancer is to return a cookie to specify which indexer to respond back to. This is the recommended way, for instance, to set up an AWS ELB for a Splunk indexer cluster such that it has cookie stickiness enabled:
https://docs.splunk.com/Documentation/AddOns/released/Firehose/ConfigureanELB https://community.splunk.com/t5/Getting-Data-In/How-to-configure-the-load-balancer-to-handle-HEC-data/td-p/742116

I'm not actually sure how acknowledgements would have worked with multiple indexers previously. From what I can tell, given the way it is structured with the hashmap keyed by ack ID, it would only ever work with single-indexer clusters due to the ack collision issue and not being able to find which indexer to properly query for acknowledgement details.

I'm also very new to writing Rust, so some of the stuff I've written might not be the best way to do things. Feedback is very welcome!

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

How did you test this PR?

I tested with some added unit and integration tests, as well as this local config and running cargo run --release -- --config test_config.yaml. This was sending to our Splunk cluster with 50+ indexers behind an AWS ALB and no events were dropped or any error logs generated over ~ 10 hours:

---
sources:
  demo_logs:
    type: demo_logs
    format: shuffle
    lines:
      - "jvperrin test log"
    sequence: true

sinks:
  splunk:
    type: splunk_hec_logs
    inputs:
      - demo_logs
    acknowledgements:
      enabled: true
      indexer_acknowledgements_enabled: true
      cookie_name: "AWSALB"
    healthcheck:
      enabled: true
    endpoint: "<REDACTED>"
    default_token: "<REDACTED>"
    index: "vector_poc"
    sourcetype: "jvperrin_test"
    encoding:
      codec: json

Does this PR include user facing changes?

• Yes. I've added a changelog entry

References

• Closes: Acknowledgement for splunk_hec_logs sink does not behave as expected when indexers are behind an LB #19417

[bits-bot]

All committers have signed the CLA.

[estherk15]

Left a small suggestion, but it's non blocking.

[estherk15]

Suggested change

	The name of a cookie to extract from the Splunk HEC response to use when querying for acknowledgements.
	Specifies the name of a cookie to extract from the Splunk HEC response and use when querying for acknowledgements.

[jvperrin]

@estherk15 Thanks for the review! I've changed this and added a bit more text describing why a cookie is useful here (specifically that it is useful for routing the request to the correct indexer)