nalu-wind spack "+tests" variant isn't working

[learndiary]

My host is ubuntu 20.04, spack command is "spack install --keep-stage nalu-wind@master%gcc@9.4.0~boost~catalyst+cuda~fftw+fsi~gpu-aware-mpi+hypre~ipo+openfast+pic~rocm~shared+tests+tioga~trilinos-solvers~umpire~wind-utils cuda_arch=61 build_type=Release ^cuda@12.5 ^openfast@master ^trilinos@master". But when I enter into its building folder execute "ctest", it reports no ctest configuration.

I edit CMakeCache.txt, changed to "ENABLE_TESTS:BOOL=ON", then execute "cmake .", only 4 of 11 tests passed show below. I want ask:
1, Is the tests of nalu-wind broking currently?
2, how to verify my nalu-wind installation is ok or not?
Thanks.

littlebat@ub20:~/tmp/spack-stage/spack-stage-nalu-wind-master-dckmaqbynd7rq7szmv4tjfie5sganq4b/spack-build-dckmaqb$ ctest
Test project /home/littlebat/tmp/spack-stage/spack-stage-nalu-wind-master-dckmaqbynd7rq7szmv4tjfie5sganq4b/spack-build-dckmaqb
      Start  1: nrel5MWactuatorLine
 1/11 Test  #1: nrel5MWactuatorLine ..............***Failed    1.94 sec
      Start  2: fsiTurbineSurrogate
 2/11 Test  #2: fsiTurbineSurrogate ..............***Failed   13.36 sec
      Start  3: airfoilRANSEdgeNGPHypre.rst
 3/11 Test  #3: airfoilRANSEdgeNGPHypre.rst ......***Failed    2.52 sec
      Start  4: ablNeutralNGPHypre
 4/11 Test  #4: ablNeutralNGPHypre ...............***Failed   11.35 sec
      Start  5: ablNeutralNGPHypreSegregated
 5/11 Test  #5: ablNeutralNGPHypreSegregated .....***Failed   27.91 sec
      Start  6: multiElemCylinder
 6/11 Test  #6: multiElemCylinder ................***Failed   19.00 sec
      Start  7: VOFZalDisk
 7/11 Test  #7: VOFZalDisk .......................   Passed    5.01 sec
      Start  8: airfoilSST_Gamma_Trans
 8/11 Test  #8: airfoilSST_Gamma_Trans ...........   Passed   42.41 sec
      Start  9: oversetRotCylNGPHypre
 9/11 Test  #9: oversetRotCylNGPHypre ............***Failed   21.93 sec
      Start 10: convTaylorVortex
10/11 Test #10: convTaylorVortex .................   Passed   35.38 sec
      Start 11: unitTestGPU
11/11 Test #11: unitTestGPU ......................   Passed    8.24 sec

36% tests passed, 7 tests failed out of 11

Label Time Summary:
regression      = 375.67 sec*proc (9 tests)
unit            =   8.24 sec*proc (1 test)
verification    =  70.76 sec*proc (1 test)

Total Test time (real) = 189.06 sec

The following tests FAILED:
          1 - nrel5MWactuatorLine (Failed)                      regression
          2 - fsiTurbineSurrogate (Failed)                      regression
          3 - airfoilRANSEdgeNGPHypre.rst (Failed)              regression
          4 - ablNeutralNGPHypre (Failed)                       regression
          5 - ablNeutralNGPHypreSegregated (Failed)             regression
          6 - multiElemCylinder (Failed)                        regression
          9 - oversetRotCylNGPHypre (Failed)                    regression
Errors while running CTest
Output from these tests are in: /home/littlebat/tmp/spack-stage/spack-stage-nalu-wind-master-dckmaqbynd7rq7szmv4tjfie5sganq4b/spack-build-dckmaqb/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

[marchdf]

Hi, thanks for reaching out.

1. The nalu-wind tests are not currently broken: https://my.cdash.org/index.php?project=Exawind#!#Nalu-Wind
2. After spack install, run spack cd -b nalu-wind then ctest
3. Your tests are running but they are reporting failures because they do not have reference gold files to compare against. Each machine/hw/os/compiler combination leads to a different set of reference files. So you first need to generate using the master branch them for your specific build, store them somewhere and then as you develop you can keep running the tests to see if you are introducing any diffs. This is the main purpose of the tests: to catch development issues. If you are not developing new features for nalu-wind, just establishing that they are running is plenty. If you are developing and needing to maintain your own reference gold files, then you can set that up with spack. The configure output, similar to https://my.cdash.org/builds/2841876, will tell you where it is saving the reference files and where it will expect them.

[learndiary]

After spack install, run spack cd -b nalu-wind then ctest

I did it after "spack install nalu-wind+tests ...", but no ctest configuration files created in the building folder. So can't execute ctest. I did the same things on exawind and amr-wind, they are both working as expected.
The main problem is it should create ctest configuration when spack install it with "+tests" variant, but it didn't.

[learndiary]

This is another "spack install nalu-wind+tests" but no ctest configuration created case:

littlebat@ub2404:~$ spack cd -b nalu-wind
littlebat@ub2404:/tmp/littlebat/spack-stage/spack-stage-nalu-wind-master-4amikb4jc7jilxpqlwpn3qvssk56ohr7/spack-build-4amikb4$ pwd
/tmp/littlebat/spack-stage/spack-stage-nalu-wind-master-4amikb4jc7jilxpqlwpn3qvssk56ohr7/spack-build-4amikb4
littlebat@ub2404:/tmp/littlebat/spack-stage/spack-stage-nalu-wind-master-4amikb4jc7jilxpqlwpn3qvssk56ohr7/spack-build-4amikb4$ ls
CMakeCache.txt       compile_commands.json  libnalu.a              Nalu-WindTargets.cmake  unit_tests
CMakeFiles           include                Makefile               naluX                   unittestX
cmake_install.cmake  install_manifest.txt   Nalu-WindConfig.cmake  src
littlebat@ub2404:/tmp/littlebat/spack-stage/spack-stage-nalu-wind-master-4amikb4jc7jilxpqlwpn3qvssk56ohr7/spack-build-4amikb4$ ctest
*********************************
No test configuration file found!
*********************************
Usage

  ctest [options]

littlebat@ub2404:/tmp/littlebat/spack-stage/spack-stage-nalu-wind-master-4amikb4jc7jilxpqlwpn3qvssk56ohr7/spack-build-4amikb4$ 

[marchdf]

@jrood-nrel do you know what might be going on here? I can't reproduce locally.

[learndiary]

I provide some more detail information:
My os is Ubuntu 20.04 in a kvm virtual machine, has a passthrough nvidia p106-100 mining card(similar to gtx1060), entire command is "spack install --keep-stage nalu-wind@master%gcc@9.4.0~boost~catalyst+cuda~fftw+fsi~gpu-aware-mpi+hypre~ipo+openfast+pic~rocm~shared+tests+tioga~trilinos-solvers~umpire~wind-utils cuda_arch=61 build_type=Release ^cuda@12.5 ^openfast@master ^trilinos@master".
When reached building nalu-wind itself, an error stopped the building:

1 error found in build log:
     36    -- Found MPI: TRUE (found version "3.1")
     37    -- CMAKE_SYSTEM_NAME = Linux
     38    -- CMAKE_CXX_COMPILER_ID = GNU
     39    -- CMAKE_BUILD_TYPE = Release
     40    -- Trilinos git commit = 73510a07
     41    -- Configuring done (9.4s)
  >> 42    CMake Error at /home/littlebat/spack/opt/spack/linux-ubuntu20.04-haswell/gcc-9.4.0/trilinos-master-kmwvq4snksgyvdh47
           owd7z5fofy3rx57/lib/external_packages/MPI/MPIConfig.cmake:17 (target_link_libraries):
     43      The link interface of target "MPI::all_libs" contains:
     44
     45        MPI::MPI_C
     46
     47      but the target was not found.  Possible reasons include:
     48

See build log for details:
  /home/littlebat/tmp/spack-stage/spack-stage-nalu-wind-master-3eoxb4pesepqiox35h6yedoai3oqpraa/spack-build-out.txt

I resolved this by add below into "trilinos-master-***/lib/external_packages/MPI/MPIConfig.cmake"

find_package(MPI REQUIRED)
add_library(MPI::MPI_C INTERFACE IMPORTED)

Then repeat that long command above again to finish the installation.

Another machine is ubuntu 24.04 with nvidia gtx 4060ti, gcc 9.5.0, the command only removed the "+fsi" variant and changed cuda_arch=86 relative to previous case, has a similar symptom.

Please tell me if need more information.