Failing test on multiple architectures

[ocaisa]

We see the same failing test for Neoverse N1, generic Arm and Haswell cpus:

368/375 Test #368: benchmark_matrix_statistics ...................................***Failed    0.09 sec
TEST: '/tmp/bot/easybuild/build/Ginkgo/1.9.0/gompi-2023b/easybuild_obj/benchmark/matrix_statistics/matrix_statistics' '-input' '[{"size": 100, "stencil": "7pt"}]'
FAIL: stdout differs
---

+++

@@ -13,7 +13,7 @@

                 "q3": 6.0,
                 "max": 7,
                 "mean": 5.8,
-                "variance": 0.7199999999999992,
+                "variance": 0.7199999999999993,
                 "skewness": -0.23570226039551892,
                 "kurtosis": 2.388888888888889,
                 "hyperskewness": -1.741577812922432,
@@ -26,7 +26,7 @@

                 "q3": 6.0,
                 "max": 7,
                 "mean": 5.8,
-                "variance": 0.7199999999999992,
+                "variance": 0.7199999999999993,
                 "skewness": -0.23570226039551892,
                 "kurtosis": 2.388888888888889,
                 "hyperskewness": -1.741577812922432,

Happy to patch that if you can give me a pointer. (Context is EESSI/software-layer#1127)

[ocaisa]

Looks like you are comparing exact output, so I may have to patch out the test given it returns different values for different architectures.

[ocaisa]

I think that just means commenting out https://github.com/ginkgo-project/ginkgo/blob/develop/benchmark/test/CMakeLists.txt#L27

[yhmtsai]

does your compiler enable some fast math or not have IEEE math operation by default? we usually assume the compiler needs to have the proper math operation from IEEE.
For example, in icpx we need to enable -ffp-model=precise, in nvhpc we need to add -Kieee to have proper result.
If you do not need the benchmark, you can turn off it by -DGINKGO_BUILD_BENCHMARKS=OFF

[ocaisa]

It's GCC 13.2.0 and CXXFLAGS set to

-O2 -ftree-vectorize -march=native -fno-math-errno -fopenmp

The same compiler and options are used for 12 different CPU architectures, 9 return the expected result and only the ones mentioned are different.

[upsj]

Does the same thing happen without -fno-math-errno? This flag sounds close enough to -ffast-math that it might also cause this behavior change.

[ocaisa]

I've tried a few things (including explicit -fmath-errno) and I can't get this test to pass.

Is this a blocker to shipping Ginkgo? I can patch out that specific benchmark very easily...

[upsj]

Not a blocker at all, but it does tell me that either the float formatting code or the actual Ginkgo code is being optimized in a likely IEEE non-compliant way, which might be an issue for people who care about exact numerical reproducibility and such. This is why e.g. we need to set the aforementioned flags for nvc++ or icpx, otherwise our tests that rely on correct treatment of NaNs or correct rounding fail.