Need to parse a yaml file using remap

[amanmahajan26]

I am relatively new to vector. I have a use case where I want to parse a yaml file and create a table out of it. I am able to do it using lua transform but did not find a way to do the same using remap. Want to know if it is possible. Thanks!

[jszwedko]

Hi @amanmahajan26 !

Could you share more about your use case? Where does the input data come from? What transformations do you want to do to it? Where do you want to send the data?

[amanmahajan26]

Hi @jszwedko, thanks for your reply. Please find the details as below:

Where does the input data come from?
The input is the logs coming from logstash.

What transformations do you want to do to it? Where do you want to send the data?
The applications that send the logs append the fields application and virtual_channel in metadata. We want to send these logs to elasticsearch. For defining the index where the logs will be stored in elasticsearch, we have the following pattern: <application>-<channel>-<date_today>. And we have a yaml config file which stores the mapping from virtual_channel to channel.

So I want to do this transformation (reading the yaml config and create a map for virtual_channel to channel) in vector using remap.

[jszwedko]

Hi @amanmahajan26 !

So this sounds like it would fit well with the concept of enrichment tables we introduced recently which allows you to load mappings from files. We only support CSV right now, but YAML seems like it'd be a good candidate to add support for. Would you want to open an issue? For now, I do think the lua transform is your best option.

[amanmahajan26]

Thanks @jszwedko , enrichment tables is what I was looking for. Support for JSON or YAML would be really a great addition to this.