Top-level @testsets stop on errors

[wavexx]

@testset should constitute rollback points for exceptions (execution should continue on the next testset), and indeed it's the case, but only for nested testsets:

using Base.Test
@testset "outer" begin
    @testset "inner" begin
        @test true
        @assert false
    end
    @testset "inner2" begin
        @test true
    end
end
@testset "next" begin
    @test true
end

Summary shows:

Test Summary: | Pass  Error  Total
outer         |    2      1      3
  inner       |    1      1      2
  inner2      |    1             1

inner2 is run, but "next" is not. It seems this is triggered by finalize(), which stops on error at the top level. Is this intended, and if so, why?

[StefanKarpinski]

I do not believe this is the intended behavior – we should fix it.

[TotalVerb]

The top-level testset has always quit upon failure. I'm not sure if it's a good thing or otherwise, but it's consistent with the behavior from 0.4 and before, where the first unwrapped @test failure quits the testsystem. Changing this would also change that.

[wavexx]

On Thu, Apr 27 2017, Fengyang Wang wrote: The top-level testset has always quit upon failure. I'm not sure if it's a good thing or otherwise, but it's consistent with the behavior from 0.4 and before, where the first unwrapped @test failure quits the testsystem. Changing this would be also change that.

I only recently started using testsets, but I can confirm this was the same in the past. However, when grouping testsets in the toplevel, this is definitely not the behavior I expect.

[dpsanders]

It would be useful to have both options available via some kind of user choice.

[wavexx]

On Fri, Apr 28 2017, David P. Sanders wrote: It would be useful to have both options available via some kind of user choice.

When would stopping be useful?

[dpsanders]

To not overwhelm you if there are too many errors, so you can fix them one by one.

[KristofferC]

When should the exception be thrown? Each testset is an individual code block.

[StefanKarpinski]

Never? It should be reported as an error in testing report and the entire test process should return non-zero to indicate that there were test failures.

[JeffBezanson]

Bumping this. Along with #25297, this is one of the key problems that makes it hard to diagnose testing issues. Get one little exception due to an unrelated problem and bam, the whole run stops and becomes useless.

[Thuener]

Another weird behavior related to this issue

using Base.Test

names = ["First", "Second"]
@testset "$(names[i])" for i = 1:2
    @test 1 == 2
end

Output:

First: Test Failed
  Expression: 1 == 2
   Evaluated: 1 == 2
Stacktrace:
 [1] macro expansion at ./untitled-acedbe26df005974be13ed82e81e5be2:5 [inlined]
 [2] macro expansion at ./test.jl:921 [inlined]
 [3] anonymous at ./<missing>:?
Test Summary: | Fail  Total
First         |    1      1
Test Summary: | Fail  Total
First         |    1      1

In this case, the First test is aborted because of the fail, seams that the Fist test is executed twice but just the message is duplicated. The Second test is never executed.