Question about Features which BullMQ Pro has (Batching, Concurrency, Sandboxed Worker and Message Handlers)

[wernermorgenstern]

Hi, we are working on a redesign of our system.
We pass messages from a Producer to consumer (Microservices).

So for example, a Producer will get a message from a MQTT Broker, and then send the message to a queue.
Then a consumer service will consume the message, decrypt it, and do other things with it, and then put it on another queue, for other microservices to consume it.

And these microservices will need to be able to scale horizontally.

However, the MQTT Publisher will send a message every 10 seconds to the MQTT Broker. So the consumers will need to be able to pick up these in order per specific Device ID (which is the MQTT Publisher).

BullMQ Pro has got the batching and concurrency feature. We can set the group ID to be the device ID, and then the consumer will only pick up one message per Group ID (which helps in that it won't consume multiple messages from the same Group ID at the same time).

Does Redis-SMQ have those features? I have not seen those features in the documentation.

I do like Redis-SMQ, especially with the performance numbers (I have not compared them myself yet, within our system). I will do that in the next few days. Also, Redis-SMQ was pretty simple to add, and replace BullMQ Pro within our code.

The afore mentioned feature in regards to Batching and Concurrency is important.

Also, BullMQ supports sandboxed worker threads within NodeJS. Does Redis-SMQ support that? Because if we can use all CPU's within our AWS EKS environment, and allow NodeJS to handle "concurrency" using worker threads, that would help a lot with resource Usage.

[weyoss]

@wernermorgenstern Thank you for opening this issue.

RedisSMQ mainly focuses on producing and consuming messages while keeping things as simple as possible. So many features that you may find in BullMQ do not exist in RedisSMQ. Otherwise it won't be RedisSMQ.

Regarding consuming one message per Group ID or consumer group ID in general, such feature does not exist but I think that it may be considered to be included in the project's roadmap in future.

Nevertheless, you may use a workaround. For example an exclusive lock per consumer based on its Group ID. The lock should be freed by the consumer which acquired it when finishing its work. Adding a lock timeout may help to automatically free it in case of a consumer crash.

Keeping this issue open for now until further updates.

[weyoss]

Pub/Sub delivery model and consumer groups are now available starting with redis-smq v8.0.0-rc.15.

[weyoss]

Message handler worker threads (aka message handler sandboxing) have been implemented and now available since redis-smq v8.0.0-rc.16.

[vorant94]

@weyoss in docs i see it is required to set groupId on consumer instance while in mentioned by topic started pro version of bullmq it is set on job instance (link)

my use case goes something like this: app has multiple users, each one having multiple events. different events of the same user should be processed in queue, different events from different users should be processed in parallel

while setting group id on job instance can allow to achieve what i seek (by setting group id to user id), does it mean that having groups on consumer level like they were introduced in v8.0.0-rc.15 doesn't fit here? because it sounds like i would have to dynamically create and delete consumer groups for each user... am I right?

P.S. just in case sorry for comparing your free open source labor with paid software, i don't mean anything rude here. i'm new to message queueing stuff and just trying to find the right tool for a job. i see there are actually two different meanings for term "group" related to mq, so its quite hard to avoid comparisons while articulating my thoughts