RabbitMQ as backend alternative?

[Kidlike]

I discovered the project from JavaOne. First of all, congratulations for the project, it does solve a lot of real problems evolved from the microservice era bloat.

So the question is, would it be possible/feasible to add an alternative to the Kafka backend, namely RabbitMQ?

I don't want to get into flame wars between RabbitMQ and Kafka, this is just about giving options.

It would make it a lot easier for large projects already using RabbitMQ to slowly migrate to LittleHorse, without having to implement/maintain/pay for a parallel Kafka cluster.

I'd be willing to contribute to make this happen.

[Snarr]

Hi @Kidlike! First off, we greatly appreciate your interest and your willingness to contribute to this open source project 😁 Thank you for opening this discussion.

We're always looking for ways to improve LittleHorse and offering alternatives to Kafka has been a recurring question.

On the topic of RabbitMQ, many of the advantages Kafka offers over RabbitMQ directly impact LittleHorse:

• Persisting data is a huge part of LittleHorse, and Kafka persists all of the messages sent through our Kernel so that it is easy to recover in the event of a failure. We can replay messages, which often represent Commands, in their correct order to restore the state of our workflows.
• Message ordering is guaranteed in single partitions, and we heavily leverage partitions to distribute work across multiple brokers for scalability.
• Scaling and high-throughput is important, as LittleHorse was designed from the ground up to be more scalable and performant than other workflow engines in the space. Kafka is an important backbone to making that possible.

There are lots of other aspects to consider here, and I'm admittedly no RabbitMQ expert. Therefore, I welcome further discussion on this, and we would greatly appreciate your contribution to the cause.

I would like to see the LittleHorse Kernel's gRPC endpoints decoupled from Kafka Streams, so that other tools, like Postgres and RabbitMQ, can be used as a backend for LittleHorse.

I know @coltmcnealy-lh and @eduwercamacaro might have more opinions on this.

Alternatives

Would it help if LittleHorse and Kafka were bundled together? We have experimented with embedding them into a single package so that they are easier to deploy into production. However, this approach will still require a Kafka cluster running somewhere but with less setup.

We also recently added LittleHorse Sink Connectors for Kafka Connect, which you may find useful down the road for migrating from RabbitMQ, as well as this page from Confluent: RabbitMQ Source Connectors for Kafka Connect.

[Kidlike]

I am not an expert either (although I've worked with both for years at different times). 😁

@Snarr As far as I can tell, RabbitMQ streams or superstreams are equivalent to Kafka. I don't know if they tick ALL the boxes you need, but superficially it looks like they should.

---

As for the alternatives... I wouldn't say bundling them together helps from operations perspective. Maybe for quick bootstrapping yes, but I see here that this is already solved. And the rmq<->kafka bridges/connectors probably introduce more complexity than solve, at least based on personal and anecdotal experience (as with all solutions that involve such connectors or adapters etc).