Surefire runOrder is ignored while running tests

[vkn]

Describe the bug

I've configured surefire plugin to execute tests in random order:

  <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>${surefire-plugin.version}</version>
                <configuration>
                    <runOrder>random</runOrder>

But the tests are always executed in alphabetical order. The surefire log however prints:
Tests will run in random order. To reproduce ordering use flag -Dsurefire.runOrder.random.seed=112036575086454

Expected behavior

The configured order should be applied

Actual behavior

The configured order is ignored

[INFO] --- surefire:3.5.0:test (default-test) @ untitled ---
[INFO] Tests will run in random order. To reproduce ordering use flag -Dsurefire.runOrder.random.seed=112036575086454
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.example.PojoATest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.147 s -- in org.example.PojoATest
[INFO] Running org.example.PojoBTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 s -- in org.example.PojoBTest
[INFO] Running org.example.PojoCTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 s -- in org.example.PojoCTest
[INFO] Running org.example.PojoDTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 s -- in org.example.PojoDTest
[INFO] Running org.example.PojoETest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 s -- in org.example.PojoETest
[INFO]

How to Reproduce?

demo.tar.gz
unpack and run mvn test multiple times

Output of uname -a or ver

No response

Output of java -version

21

Quarkus version or git rev

3.17.3

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.9.9

Additional information

No response

[geoand]

Thanks for reporting.

This actually sounds more like an integration issue between JUnit5 and Maven surefire. In Quarkus we simply implement org.junit.jupiter.api.ClassOrderer which should be agnostic of how tests were launched.

[vkn]

Thanks for reporting.

This actually sounds more like an integration issue between JUnit5 and Maven surefire. In Quarkus we simply implement org.junit.jupiter.api.ClassOrderer which should be agnostic of how tests were launched.

When I remove quarkus dependencies it works as expected

[geoand]

Can you please attach a sample showing that?

[vkn]

@geoand
Without quarkus: no_quarkus.tar.gz
With quarkus: demo.tar.gz

[geoand]

I checked and this is not a Quarkus specific issue.
If the application contains a JUnit 5 org.junit.jupiter.api.ClassOrderer, then the surefire configuration is pointless as the JUnit orderer is executed later