Enhancing our Native Performance Monitoring

[cescoffier]

Hello,

Last week, @jerboaa and some team members (including myself) discussed native testing and how we can improve it. I’d like to continue that discussion and define the set of “tests” (repositories, test cases, etc.) we use to continuously monitor our performance in native mode.

For each test, the measurements should include:

• Startup RSS
• Time to first request (when applicable)
• RSS after the first request
• Disk footprint (even though we haven’t closely monitored it)
• Reachability reports
• Additional metrics as needed

These measurements aim to catch regressions in Quarkus and anticipate changes in GraalVM/Mandrel.

Currently, we are running performance tests for:

• A simple application using Picocli
• An application using the OpenShift Client
• The (infamous) “main” IT

I believe this set needs to be updated:

1. I’m not sure “main” is very useful.
2. Having more “real-life” examples would provide more valuable insights. Something like the Super Heroes “Villain” service (Quarkus REST + Hibernate ORM + Panache) would be very interesting, as it represents a common type of application.
3. The Hibernate ITs from Quarkus Core, such as jpa-postgresql and jpa-postgresql-withxml, would help us track performance changes when Hibernate is updated (as it can significantly impact metrics).
4. More complex but realistic examples, such as the Super Heroes “Fights” service, could also be valuable (Kafka, REST Client, Avro, Mongo, Stork, gRPC).

Point (4) might be too complex to start with, but including (2) — a typical CRUD application — would be very useful.

WDYT? How would you like to go forward?

\CC: @jerboaa @galderz @zakkak @franz1981 @radcortez @Sanne

[jerboaa]

Our current Mandrel testing is hooked into GHA testing where we run the quarkus native integration tests. So removing openshift client and main from the tracking and adding jpa-postgresql and jpa-postgresql-withxml seems easy enough to do. For point (2) I'm not sure how easy that one is to set up in GHA and track the various Mandrel for JDK X release trains (all using quarkus main). Currently we test Mandrel for JDK 21 (internal version 23.1), Mandrel for JDK 24 (internal version 24.2) and Mandrel for JDK 25 (internal version 25). We'll have to take a look.

To me it sounds like we should approach this in a staged approach:

1. Move the set of tested apps from picocli, main, openshift client to picocli, jpa-postgresql, jpa-postgresql-withxml as a first step
2. Evaluate the Super Heroes service for inclusion in our testing and get that added.
3. See how we can go from there.

FWIW, the reason why we've picked openshift-client extension for testing was that it was notoriously reflection heavy in the past (which I think is still the case).

My $0.02 :)

[zakkak]

Currently, we are running performance tests for:

• A simple application using Picocli
• An application using the OpenShift Client
• The (infamous) “main” IT

Please note that the "performance tests" for these tests/benchmarks that we run are regarding build-time only metrics, i.e.:

• Disk footprint -- which is kind of coupled with Reachability reports
• build-time RSS usage

For run-time performance monitoring we use:

• getting-started
• config-quickstart

from https://github.com/quarkusio/quarkus-quickstarts and monitor the following metrics:

• Startup RSS
• Warm RSS (after multiple requests)
• Throughput

while we collect data for

• RSS after the first request

but not

• Time to first request (when applicable)

@franz1981 can you please confirm?

I believe this set needs to be updated:

1. I’m not sure “main” is very useful.

I find main useful in the sense that is the biggest test, bringing in most dependencies. I agree that it's probably not worth putting much effort in optimizing, and that it is hard to attribute regressions to specific parts of it, but I still find it useful for the sake of having more coverage, at least for build-time metrics.

2. Having more “real-life” examples would provide more valuable insights. Something like the Super Heroes “Villain” service (Quarkus REST + Hibernate ORM + Panache) would be very interesting, as it represents a common type of application.

+1

3. The Hibernate ITs from Quarkus Core, such as jpa-postgresql and jpa-postgresql-withxml, would help us track performance changes when Hibernate is updated (as it can significantly impact metrics).

+1, as @jerboaa says the switch to (or just addition of) these tests should be trivial. I will have a look for build-time metrics.

4. More complex but realistic examples, such as the Super Heroes “Fights” service, could also be valuable (Kafka, REST Client, Avro, Mongo, Stork, gRPC).

+1

How would you like to go forward?

I agree with @jerboaa's steps and will start working on 1 (but only for build-time metrics, the run-time part requires more effort at this time)

[jerboaa]

Thanks for the clarifications, @zakkak. Agreed to all points mentioned.

[cescoffier]

About the villain, what about having an IT in quarkus core "replicating" the same kind of application? Can it be a quickstart? (to simplify the onboarding)

[jerboaa]

Having it as a quickstart or IT in quarkus would help, I think, yes.

[franz1981]

@zakkak

but not
Time to first request (when applicable)

Yep it doesn't look like it is checked - whilst https://github.com/franz1981/quarkus-startup/blob/master/drivers/loop.yaml does it instead.

[cescoffier]

Having it as a quickstart or IT in quarkus would help, I think, yes.

Alright, then I believe we could have "villain" very quickly.

[cescoffier]

What would be the next natural step to get this in motion?

[jerboaa]

Native IT would work best for us.

[zakkak]

@cescoffier FYI I have already setup monitoring of build-time metrics of jpa-postgresql and jpa-postgresql-withxml for about a week now (and I also included data since June 2024), just didn't update here since I am also waiting for some automation to be enabled.

IT or quickstart. Which is most convenient for you?

Native IT would work best for us.

Although IT is indeed more convenient for us (mandrel team) since we already have the setup for collecting build-time metrics from other ITs, I believe a quickstart would be more convenient for run-time metrics. For run-time metrics we would need to run the tests against different Quarkus versions and I don't think ITs are convenient in that manner.

WDYT @radcortez @franz1981 ?

[cescoffier]

@zakkak @jerboaa - I've created quarkusio/quarkus-quickstarts#1523 - Let me know what you think.

[zakkak]

@cescoffier just a note that I have seen this comment but have still not found the time to go through the PR

[cescoffier]

No problem @zakkak