Log namespace configuration for sink

[piclemx]

Hello,

I'm trying to make my configuration of vector that would contains the following.

I want to use the kafka.topic, kafka.partition and kafka.offset to populate the field of sourcetype with something like

Should result into this topic::partition::offset

I have enable the log_namespace to true also globally.

Source:

• Kafka

Sink:

• Splunk HEC Logs

[pront]

Preserving discord discussion here:

piclem — 11:44
What would be the way to get the source metadata from the remap

Pavlos — 11:45

event = .
field_from_event = .foo

all_metadata = %
tags = %datadog_agent.ddtags
timestamp = %vector.ingest_timestamp

Sample from https://vector.dev/blog/log-namespacing/

Namespace feature guide: https://vector.dev/guides/level-up/log_namespace/

[piclemx]

Ok, I found the first of the problem.

the @timestamp is on the root itself.

."@timestamp" this return the right value now.

I added this part

all_metadata = %
log(all_metadata, level: 'info')

and I saw the the kafka input metadata isn't like the example you post

Internal log [{"kafka":{"headers":{},"message_key":null,"offset":168906310,"partition":0,"timestamp":"2025-05-21T13:53:41.771Z","topic":"topic"},"vector":{"ingest_timestamp":"2025-05-21T17:50:51.595688Z","source_type":"kafka"}}]

[pront]

The kafka sink will attempt to read the timestamp from the path specified in set_semantic_meaning.

From the log sample above you probably want:

set_semantic_meaning(%kafka.timestamp, "timestamp")

Again, this might require more trial and error.

[piclemx]

I'm not using the kafka sink. I'm using the Splunk HEC Logs sink.

which is still complaining about it.

2025-05-21T17:56:09.292648Z  WARN sink{component_kind="sink" component_id=splunk_out component_type=splunk_hec_logs}: vector::internal_events::splunk_hec::sink: Timestamp was an unexpected type. Deferring to Splunk to set the timestamp. invalid_type="string" internal_log_rate_limit=true

[pront]

You are hitting the following:

vector/src/sinks/splunk_hec/logs/sink.rs

Lines 298 to 300 in 34db5b0

	emit!(SplunkEventTimestampInvalidType {
	r#type: value.kind_str()
	});

.

This means your timestamp value is not Value::Timestap, it is a string as shown in the sample log above "timestamp":"2025-05-21T13:53:41.771Z".

See example on how to convert timestamp string: example.

This is covered in the VRL introduction page: https://vector.dev/docs/reference/vrl/.

[piclemx]

Yeap, I found and and adjust my remap and seems to work now.

last thing, Is there a way to this to add a % metadata field and can be used where https://vector.dev/docs/reference/configuration/sinks/splunk_hec_logs/#source ?

[piclemx]

."@timestamp", _ = parse_timestamp(."@timestamp", format: "%Y-%m-%dT%H:%M:%S.%fZ")
 set_semantic_meaning(."@timestamp", "timestamp")

this should be the answer.

[piclemx]

Ok, weird things. I'm adding a field to the given event.

          topic = to_string(%kafka.topic) ?? "topic"
          partition = to_string(%kafka.partition) ?? "partition" 
          offset = to_string(%kafka.offset) ?? "offset"
          .source = "kafka::{{ topic }}::{{ partition }}::{{ offset }}"

this appears after i'm doing a "log"

but when I look at the sink result in Splunk The field isn't there.