VRL emitting multiple events loses event metadata?

[sbalmos]

When a VRL transform emits multiple events, it does not copy over, or otherwise include at all, any event metadata. Subsequent transforms and sinks lose access to the source_type, timestamp, and other Vector-generated fields, along with metadata that is generated from the source. Is there a reason for this?

Currently now, I have to do something like the following to basically "reconstruct" the metadata in a VRL transform before emitting the multiple events. For reference, this is when reading Azure Event Hub events using a Kafka source. The events are batched by Event Hub into a records array.

metadata = .
del(metadata.message)
records = parse_json!(.message).records
records = map_values(array!(records)) -> |record| {
  entry = {}
  entry |= metadata
  entry.message = record
  entry
}
. = records

All so I can maintain the original event structure when emitting the multiple values.

[sbalmos]

This feels weird, but I guess this is what unnest() really is for? I've sort of reduced it down to the following now...

.message = parse_json!(.message).records
. = unnest!(.message)

Note that if you did

.message = parse_json!(.message)
. = unnest!(.message.records)

You'd end up with multiple events properly, but message would have a single child object records that then contained the record data.