Streaming ingestion and max file size

[i3arnon]

The Java client will support streaming ingestion in the near future.
This allows this plugin to support it as well.
So, we can add an optional configuration for streaming.

However, streaming has a size cap (4mb), which AFAIK the plugin doesn't yet have.
So, I suggest also adding an optional configuration for max file size, which when reached creates a new file with some suffix and when streaming would be turned on that max size will be be the min of the request max size and 4mb.

@tamirkamara WDYT?

[tamirkamara]

@i3arnon
I think that is a possibility, although I'm not sure how you'll keep watch over the file size and know which suffix to add to each event...
Remember that currently the rolling mechanism is based on the event time and some events might get processed by the output plugin later despite them having an earlier timestamp.

Depending on the effort you plan to invest here, perhaps think about using java streams which should give better e2e performance for the streaming ingestion scenario.
What do you think?

[vladikbr]

Taken as a feature request.