`#[tokio::test]` functions are mutated

[fence-sitter]

When testing async code I use the #[tokio::test] annotation instead of #[test]. This simplifies the tests because I do not need to manually configure a runtime. However cargo-mutants does not recognise that these are test functions and proceeds to mutate them.

Cargo.toml:

[package]
name = "cargo-mutants-tokio-test"
version = "0.1.0"
edition = "2024"

[dependencies]
tokio = { version = "1.41.1", features = ["macros", "rt"] }

src/lib.rs:

#[test]
fn test() {
    println!("Hello")
}

#[tokio::test]
async fn tokio_test() {
    println!("Hello")
}

Terminal:

$ cargo install --git "https://github.com/sourcefrog/cargo-mutants" --locked
    Updating git repository `https://github.com/sourcefrog/cargo-mutants`
     Ignored package `cargo-mutants v25.3.1 (https://github.com/sourcefrog/cargo-mutants#49940940)` is already installed, use --force to override
$ cargo mutants
Found 1 mutant to test
ok       Unmutated baseline in 3.6s build + 0.1s test
 INFO Auto-set test timeout to 20s
MISSED   src/lib.rs:8:5: replace tokio_test with () in 0.3s build + 0.1s test
1 mutant tested in 4s: 1 missed

[fence-sitter]

Workarounds include:

• Use #[test] and manually configure the runtime.
• Use #[cfg(test)] as well as #[tokio::test] (as per src/visit.rs#L802)

[fence-sitter]

Although the fix for this specific annotation seems simple enough, is this something that should actually be supported?

• Workarounds exist and are not that obtrusive.
• Where do you draw the line when it comes to supporting other test annotations?
  • Eg. I also use #[sqlx::test]. Should that be supported as well?
  • Should an attempt be made to catch all attributes that look like #[whatever::test]?

[sourcefrog]

I have a general principle (https://mutants.rs/goals.html) that it should do something reasonable on most Rust trees without needing any setup. Since Tokio is very well known matching on #[tokio::test] by default makes sense; sqlx::test probably makes sense, even probably whatever::test.

We could also add a configuration option for attributes to skip.

However also I'd say your code is a bit unidiomatic in not wrapping tests in #[cfg(test)]: https://doc.rust-lang.org/book/ch11-03-test-organization.html#the-tests-module-and-cfgtest

[fence-sitter]

However also I'd say your code is a bit unidiomatic in not wrapping tests in #[cfg(test)]: https://doc.rust-lang.org/book/ch11-03-test-organization.html#the-tests-module-and-cfgtest

Thank you for this! I had no idea that #[test] functions were compiled during cargo build.

I've put my tests inside dedicated modules, so now my builds are faster and this issue is no longer relevant to me. I'll leave this for you to close (or leave open, if you think it should be addressed).

---

For reference, the original example with tests in modules.

src/lib.rs:

#[cfg(test)]
mod test {
    #[test]
    fn test() {
        println!("Hello")
    }

    #[tokio::test]
    async fn tokio_test() {
        println!("Hello")
    }
}

Terminal:

$ cargo install --git "https://github.com/sourcefrog/cargo-mutants" --locked
    Updating git repository `https://github.com/sourcefrog/cargo-mutants`
     Ignored package `cargo-mutants v25.3.1 (https://github.com/sourcefrog/cargo-mutants#49940940)` is already installed, use --force to override
$ cargo mutants
Found 0 mutants to test
 WARN No mutants found under the active filters

[kpreid]

I had no idea that #[test] functions were compiled during cargo build.

This is not true. If it were, the following program would not compile in non-test mode, but in fact it does.

#[test]
fn foo() { bar(); }
#[cfg(test)]
fn bar() {}

The book appears to be wrong on this point. The real reason why #[cfg(test)] mod tests is so common is that a module of tests very often has some associated uses, if not other items such as shared helper functions, which would be dead code (and thus provoke warnings) except when running tests. It also might be very slightly faster to build (because the compiler only needs to process one cfg instead of many).

[sourcefrog]

@kpreid Thanks. Do you have an opinion on skipping functions with an attribute whose path ends in test?

[kpreid]

I think it’s a reasonable default. It’s the sort of thing that I can imagine someone wanting to turn off, though. Maybe there should be a config option for “disable all heuristics and filter only in ways I specify”?