Performance regression when using Hibernate with reactive message queues

[LarsSven]

We designed a Quarkus service that makes use of REST, Hibernate ORM and messaging. We deliberately decided to step away from mutiny so we are not using that here. There is a very specific guide page in the documentation: https://quarkus.io/guides/kafka#writing-entities-managed-by-hibernate-to-kafka

This is essentially our use case. It is of course a bit more complex/larger, but it essentially boils down to:

1. Receive message by REST
2. Store entity in DB using Hibernate
3. Send entity DTO over message queue
4. Update entity status in DB to QUEUED

The documentation suggests this:

@Path("/")
public class ResourceSendingToKafka {

    @Channel("kafka") Emitter<Fruit> emitter;

    @POST
    @Path("/fruits")
    @Transactional                                                      
    public CompletionStage<Void> storeAndSendToKafka(Fruit fruit) {     
        fruit.persist();
        return emitter.send(new FruitDto(fruit));                       
    }
}

However this creates an annoying performance regression for us (which took us some time to debug):
Transactions need to be committed at the end, however, since we now have an async emitter.send that returns a completionstage, the transaction commit now happens on the async emitter thread rather than the normal worker threads that you get with @Blocking. There is however only 1 such thread so all commits suddenly have to be handled sequentially on one thread. So by having the message queue within the transaction, our requests per minute slowed down from 13k to 8k (and likely much more as we scale the application) since all commits now happen on a single thread, even if we remove step 4 and don't do any Hibernate actions within the callback. How are we supposed to deal with this such that we don't lose this performance, especially since the guide seems to suggest this is the way to deal with this, hiding the fact that adding a message queue drops your transaction commit throughput.

[quarkus-bot[bot]]

/cc @DavideD (hibernate-reactive), @gavinking (hibernate-reactive)

[LarsSven]

We are using the non-reactive version of Hibernate, Hibernate ORM

[cescoffier]

Try with:

@Path("/")
public class ResourceSendingToKafka {

    @Channel("kafka") MutinyEmitter<Fruit> emitter;

    @POST
    @Path("/fruits")
    @Transactional                                                      
    public void storeAndSendToKafka(Fruit fruit) {     
        fruit.persist();
        return emitter.sendAndAwait(new FruitDto(fruit));                       
    }
}

Or, if you don't care about the Kafka emission (and don't want to wait for the ack from the broker) you can either:

• Set Kafka ack to 0
• Or use:

@Path("/")
public class ResourceSendingToKafka {

    @Channel("kafka") MutinyEmitter<Fruit> emitter;

    @POST
    @Path("/fruits")
    @Transactional                                                      
    public void storeAndSendToKafka(Fruit fruit) {     
        fruit.persist();
        return emitter.sendAndForget(new FruitDto(fruit));                       
    }
}

[LarsSven]

This indeed worked! We did care about the ack so we used sendAndAwait, which got our performance back to 13k RPM. Thanks a lot for the help!

2 questions:

1. Why does the MutinyEmitter support this, but not the normal Emitter?
2. Would it maybe be a good idea to add this to the documentation for Kafka & Hibernate ORM (https://quarkus.io/guides/kafka#writing-entities-managed-by-hibernate-to-kafka)? I would be willing to make a PR with some documentation updates for this, but I am curious if you think it may be useful to explain this possible performance pitfall in the documentation @cescoffier?

[cescoffier]

About Emitter vs MutinyEmitter:

The Emitter API is defined by the reactive messaging spec, so we follow the API strictly. Unfortunately it means there are things we cannot do. You can do something like toCompletableFuture().join().

The MutinyEmitter is way more optimized and should be the standard choice (so +1 about updating the doc).

CC @ozangunalp

[ozangunalp]

We need to change that example in the docs to use MutinyEmitter#sendAndAwait. Returning CompletionStage there doesn't make sense.

[LarsSven]

I have a PR open: #45819 (comment)

[LarsSven]

One solution would be to do something like:

@Path("/")
public class ResourceSendingToKafka {

    @Channel("kafka") Emitter<Fruit> emitter;

    @POST
    @Path("/fruits")
    @Blocking                                                      
    public CompletionStage<Void> storeAndSendToKafka(Fruit fruit) {     
        store(fruit)
        return emitter.send(new FruitDto(fruit));                       
    }

    @Transactional    
    private void store(Fruit fruit) {
        fruit.persist();
    }
}

But this goes against the Quarkus recommendation of putting @Transactional on the resource, makes it impossible to do database operations after the message queue operation (which may be a limitation that is impossible to work around), and is not clear from the documentation that this makes a significant performance difference.