tests fail

[flexie]

I ran the tests by running

(@v1.11) pkg> test InvertibleNetworks

resulting in

Test Summary: | Pass Fail Total Time
Test test_layers/test_layer_conv1x1.jl | 28 1 29 1.0s
Test test_layers/test_coupling_layer_irim.jl | 10 1 11 0.3s

Did these always fail? Only by increasing the tolerances by a large amount makes removes this error. I suspect there are some bugs in the code.

[grantbruer]

Furthermore, the tolerance on some of these gradient tests is so high that saying the gradient is zero would pass the tests. Details below.

The ones that look like this typically have 5 iterations of with successively halved step size, so the error without gradient information should be decreased by a factor of 1/16 (convergence order 1) and the other by 1/256 (convergence order 2). The tolerance means that this tests that the factor decreases are < 11/16 (convergence order 0.13) and < 11/256 (convergence order 1.13).

@test isapprox(err3[end] / (err3[1]/2^(maxiter-1)), 1f0 + 1f1; atol=1f1)
@test isapprox(err4[end] / (err4[1]/4^(maxiter-1)), 1f0 + 1f1; atol=1f1)

But some have only 4 iterations, so the factors should be 1/8 and 1/64. The tolerance means that this `tests that the factor decreases are < 11/8 (convergence order -0.15) and < 11/64 (convergence order 0.84), meaning the seconder order convergence test could be passed without the gradient information.

[grantbruer]

Since it is relevant to my research, I deeply investigated the gradients for the Glow network and its components.

1. If the activation function used by the ResBlock in a Glow layer is noninvertible, the code silently gives incorrect results.

• This is an issue because the default activation function is ReLU (non-invertible).

2. ExpClampLayer has incorrect forward computation.

3. For gradient tests involving Conv1x1, the gradients often have difficulty passing the gradient test. This is due to the floating-point error introduced in the forward and inverse functions. I see gradient robustness when I switch to 64-bit floats.

4. Some tests had difficulty passing due to ReLU not being differentiable at 0.

5. For test_coupling_layer_irim.jl, it looks like the step size used in the gradient test was too small.
   The test passes if I make it a little larger.

[grantbruer]

#123 fixes problems 1, 2,4, and 5 from my previous comment.