Check for possible silently passing invalid tests by making sure some sort of `panic!` exists in tests

[kayagokalp]

What it does

Consider the following code piece:

...
#[test]
fn test_with_silent_fail() {
  let result = ...;
  
  if result.is_err() {
     eprintln!("failed to do something");
  }
}

This is recently discovered for a timout use case, the test had fail cases correctly implemented but also had a outer check for timeouts with tokio runtime. If the user is using eprintln! inside a test, I feel like the test should also fail in that block. Otherwise println! should be used instead.

So in the control graph, the block that contains eprintln should contain a panic as well, or eprintln should be a println in my opinion and clippy can enforce this.

Advantage

It helps preventing cases where a test can silently pass, while the intention was to give an error thus fail.

Drawbacks

It might hurt the expressivity a little, as people may want to use eprintln inside a test for other reasons than printing an error.

Example

#[test]
fn test_with_silent_fail() {
  let result = ...;
  
  if result.is_err() {
     eprintln!("failed to do something");
  }
}

Could be written as:

#[test]
fn test_with_silent_fail() {
  let result = ...;
  
  if result.is_err() {
     panic!("failed to do something");
  }
}

or

#[test]
fn test_with_silent_fail() {
  let result = ...;
  
  if result.is_err() {
     println!("failed to do something");
  }
}

[Centri3]

We could probably generalize this to tests that never actually fail (modulo println! failing or someting).

[kayagokalp]

We could probably generalize this to tests that never actually fail (modulo println! failing or someting).

Yeah makes sense to me!

[kayagokalp]

@rustbot claim

[kayagokalp]

I got a reference implementation here at: #13579 Basically a generic implementation checking if a test can fail or not. If a test is not possible to panic it is flagged.