how to let through only the given fields

[tshepang]

Say you have an event with unknown number of fields, and are only interested in some of them that you know of. How do you discard the rest?

[tobz]

This would depend on where you want to do the field filtering.

If you wanted the events to be cleaned up after being ingested via their source, you could use the remap transform with a program such as this:

# We can create a new object by using the `filter` function to only keep fields we know about.
# We'll pretend that `unfiltered_object` is our event object. Substitute this with `.`, etc, as
# needed to apply it to your own situation:
unfiltered_object = { "a": "true, "b": true, "aa": false }
known_fields = ["a", "b"]

filtered_object = filter(unfiltered_object) -> |key, _value| {
    # Only keep fields that are in `known_fields`.
    includes(known_fields, key)
}

# Now `filtered_object` will only contain fields that are part of `known_fields`, so `a` and
# `b`, based on the above code.

If you needed to keep those unknown fields around but remove them specifically for a given sink, you could use the encoding.only_fields configuration setting to specify which event fields should be encoded, while ignoring the rest. Nearly all sinks that I know of should support that configuration setting, I believe, but here's a link to the setting on the aws_s3 sink as an example: https://vector.dev/docs/reference/configuration/sinks/aws_s3/#encoding.only_fields

[jszwedko]

Also relevant: issue to add a function to VRL, https://github.com/vectordotdev/vector/issues/7347.