Splunk HEC Sink: Set Host From Field But Not Transmit in Payload?

[tommyorndorff]

I'm receiving and shipping some data from a Datadog lambda extension.

Sample data:

[
    {
        "message": {
            "message": "2024-12-01 17:54:22 UTC | DD_EXTENSION | DEBUG | Runtime done metrics: {responseLatency:0.947 responseDuration:0.091 producedBytes:35}\n",
            "lambda": {
                "arn": "arn:aws:lambda:us-west-2:123:function:some-function",
                "request_id": "a941db01-99f3-45c6-923f-1133a3e055a1"
            }
        },
        "status": "info",
        "timestamp": 1733075662661,
        "hostname": "arn:aws:lambda:us-west-2:123:function:some-function",
        "service": "dd-lambda-playground",
        "ddsource": "lambda",
        "ddtags": "functionname:some-function,aws_account:123"
    }
]

A http server input, simple transform and simple output is fine (a bit confusing but fine). I can set the host field just fine using the sink after transforming:

[sources.http]
type = "http_server"
address = "0.0.0.0:8282"
encoding = "json"
path = "/api/v2/logs"
method = "POST"

[transforms.set_splunk_fields]
type = "remap"
inputs = ["http"]
source = '''
.host = .hostname
del(.hostname)

.ddtags = parse_key_value!(.ddtags,key_value_delimiter: ":",field_delimiter: ",")
.account_id = .ddtags.account_id
'''

[sinks.splunk_hec_logs]
type = "splunk_hec_logs"
inputs = ["set_splunk_fields"]
endpoint = "https://splunk-hec"
encoding.codec = "json"
default_token = "123"
sourcetype = "datadog:vector:{{ sourcetype }}"
index = "main"
tls.verify_certificate = false
timestamp_key = "timestamp"

But how do I remove the .host field from the Splunk payload? That field is already indexed; theres no need to pass it in the body. But if I remove it in the transform, then I cant set it in the sink...

[tommyorndorff]

I suspect my problem is something to do with log namespacing and the warning referenced. Ill run through that, see if I get further and report back.

[pront]

If you set:

schema:
  log_namespace: true

Then your events will be shaped differently. In this case, the implementation will look for the timestamp meaning, usually in %timestamp. But the key here is the % prefix, which points to the metadata root.

Also, I suppose you also want auto_extract_timestamp to be false.

One more thing to check is timestamp_key and also, the global schema options.

[tommyorndorff]

That makes sense, thank you.

In this case, my sink is http_server. I need to move key:values into %http_server (for example, %http_server.account_id). Can I later specify this %http_server.account_id as an indexed field?

Right now when I try this I receive a Configuration error. error=Invalid field path "%http_server.account_id". The relevant portion of the output is:

[sinks.splunk_hec_logs]
type = "splunk_hec_logs"
inputs = ["set_splunk_fields"]
...
indexed_fields = [
    "%http_server.account_id",
    "architecture",
    "functionname",
    ]

If I could do that, I could add the low cardinality tags into the indexed fields, then pass only the .message as the event body.

[pront]

What you want to ensure is that the paths in indexed_fields are valid event paths. The sink processes each log and attempts to get the value for each path, see here. Note that %http_server points to the http_server metadata root.