Skip to content

SONARJAVA-5410 Modify rule S5977: Improve Rationale for Avoiding Randomness in Tests #4950

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 22, 2025
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 20 additions & 10 deletions rules/S5977/java/rule.adoc
Original file line number Diff line number Diff line change
@@ -1,17 +1,27 @@
== Why is this an issue?

Tests should always:

* Make sure that production code behaves as expected, including edge cases.
* Be easy to debug, i.e. understandable and reproducible.

Using random values in tests will not necessarily check edge cases, and it will make test logs a lot harder to read. It is better to use easily readable hardcoded values. If this makes your code bigger you can use helper functions.
Randomness in test code, whether introduced intentionally to cover multiple scenarios or unintentionally through non-deterministic library functions, undermines the principles of effective testing.
In most cases, randomness leads to problems, resulting in code that is unreliable and difficult to debug.
Consequently, deterministic and reproducible tests are preferred, primarily for the following reasons:

* When a test fails, the ability to reproduce the conditions that led to the failure is crucial for effective debugging.
Randomness can make it difficult or even impossible to pinpoint the root cause, as subsequent runs may not exhibit the same failure.
* Being able to replay a scenario allows us to easily compare logs between different test runs.
* Determinism gives us confidence that a bug is fixed when it no longer appears in tests.
If they behave randomly, a passing test after a fix might be coincidental due to a specific random input, rather than a genuine resolution of the underlying problem.
* Flaky tests, which pass or fail intermittently without any code changes, are a significant problem for CI pipelines (continuous integration).
They erode confidence in the CI system, lead to unnecessary investigations and reruns, and ultimately slow down the development and release process.
A stable CI pipeline relies on deterministic test outcomes.

This rule raises an issue when ``++new Random()++`` or ``++UUID.randomUUID()++`` are called in test code.

There is one valid use case for random data in tests: when testing every value would make tests impractically slow. In this case the best you can do is use random to test every value on the long run. You should however make sure that random values are logged so that you can reproduce failures. Some libraries exist to make all this easier. You can for example use property-based testing libraries such as https://github.com/jlink/jqwik[jqwik].
== How to fix it

* When a test uses random numbers to generate inputs, an easy fix is to replace those random inputs with pseudo-random values generated from a known seed.
By initializing a pseudo-random number generator with a fixed seed, tests can generate sequences of seemingly random data that are reproducible across different test runs.

This rule raises an issue when ``++new Random()++`` or ``++UUID.randomUUID()++`` are called in test code.
* When randomness occurs due to the use of a library function, the solution is to replace the call with a constant.
For example, rather than generating a UUID at random, one should use a fixed value.


=== Noncompliant code example
Expand All @@ -27,15 +37,15 @@ UUID userID = UUID.randomUUID(); // Noncompliant

[source,java]
----
int userAge = 31;
static final int SEED = 0x533d;
int userAge = new Random(SEED).nextInt(42);
UUID userID = UUID.fromString("00000000-000-0000-0000-000000000001");
----


== Resources

* https://phauer.com/2019/modern-best-practices-testing-java/#use-fixed-data-instead-of-randomized-data[Modern Best Practices for Testing in Java - Philipp Hauer]
* https://jqwik.net/[Jqwik test engine]



Expand Down