Parquet Output

[satellite-no]

I know Vector does not support parquet format out of the box but I was doing some research and I think it would be possible to use a protobuf to encode the output object into the parquet format. I might be completely off base here but was curious if someone has attempted or had any level of success with getting parquet out of Vector.

[bfung]

There's some prior discussion and work:

• https://github.com/vectordotdev/vector/issues?q=is%3Aissue%20parquet
  • in particular: Support parquet columnar format in the aws_s3 sink #1374
• https://github.com/vectordotdev/vector/pulls?q=is%3Apr+is%3Aopen+parquet

[satellite-no]

Hey thanks far the reply! Yep Im aware of the issue and high demand far this feature I was specifically curious if anyone found a workaround using protobuf? I didnt see this in what you linked unless i missed it? Thanks again.

[pront]

I haven't seen any discussions on this. What you described should be doable because parquet files can be represented by protobufs and Vector can encode those but it ultimately depends on the downstream consumer.

[satellite-no]

Hey, im going to look deeper into this. My consumer would just be s3 so pretty opened as far as what it can consume lol. I'll report back with any success.