Add headers to the CSV output

[terebentina]

Hi all,

I have this requirement to get data from kafka and save it to s3 into csv files.
While waiting for #7124 I tried my hand with remap to convert from json to csv lines and this works. However, I'd like to add the headers on the first line of each file and I have no idea how so...help, please!

This is where I got to:

keys = ["a", "b"]
values = []
for_each(keys) -> |idx, val| {
  values = push(values, to_string!(get!(., [val])))
}
.message = "\"" + join!(values, separator: "\",\"") + "\""

Given 2 events like

{ "a": "foo1", "b": "bar1" }
{ "a": "foo2", "b": "bar2" }

this produces the output

"foo1","bar1"
"foo2","bar2"

What I'd like to do is:

"a","b"
"foo1","bar1"
"foo2","bar2"

for each saved file.
Any ideas?

[jszwedko]

Hey @terebentina !

Unfortunately achieving that in Vector will be difficult at the moment. I think your best option will likely to be to post-process the files to add that header.

An enhancement for CSV output is being tracked by #7124