Add support for RabbitMQ streams

[dantamsdb]

RabbitMQ 3.9 has added a new feature called streams. See here: https://www.rabbitmq.com/streams.html

As described on the above page, one of the advantages of RabbitMQ streams versus queues is throughput performance.

If Logstash added support for RabbitMQ streams both as input and output, then Logstash users who integrate with RabbitMQ, and who are able to switch from queues to streams, would get much better throughput performance.

Today we are using RabbitMQ queues as both Logstash input and output, and we are not happy with the performance.

[daniel-bray-sonalake]

Are there asny plans to add streams support to the backllog?

I can use this plugin to process rabbit events on a stream, but I can't find a way to set the offset. This means that any messages written to the stream while logstash is down are lost.

I've raised this here to see if it's a config issue on my part: https://discuss.elastic.co/t/can-the-logstash-rabbitmq-input-plugin-read-from-the-start-of-a-stream/296084

But if the plugin can't support streams at all, and won't for the forseable future, we'll have to find another approach.

Thanks

[deepakLabs]

Agree with @dantamsdb, @mashhurs Can we include this as well inn the sprint? This is a significant upgrade over queues and will boost the usability and performance of RabbitMQ based Logstash by miles.
cc :
@dantamsdb , @daniel-bray-sonalake

[bonobo78]

Hi,
I was reading RabbitMQ documentation on streams and it appears that it is supported but only with stream core see the stream core and stream plugin comparison page.

=> An AMQP 0.9.1 client library that can specify optional queue and consumer arguments will be able to use streams as regular AMQP 0.9.1 queues.

The problem with the consuming process is that you have to fix the x-stream-offset value to control where to start in the stream and the only reasonable and permanent values are next or last.

It is not clear for me, but it seems that RabbitMQ stores the last offset consumed for every clients.
Then with x-stream-offset set to next, it makes no changes for Logstash with classic queues.

_Fred.