OpenMP test link failed

[VictorEijkhout]

I'm building with

cmake -D CMAKE_INSTALL_PREFIX=/scratch/00434/eijkhout/installation/installation-ginkgo-1.10.0-stampede3-gcc13.2-omp-omp/omp -Wno-dev                          -DCMAKE_COMPILE_WARNING_AS_ERROR=OFF             -D CMAKE_POLICY_VERSION_MINIMUM=3.13             -D CMAKE_VERBOSE_MAKEFILE=ON             -D BUILD_SHARED_LIBS=ON             -D CMAKE_BUILD_TYPE=RelWithDebInfo             -DGINKGO_BUILD_OMP=ON   /work2/00434/eijkhout/ginkgo/ginkgo-1.10.0/

and the OpenMP flag is correctly inserted during compilation:

[  9%] ESC[32mBuilding CXX object omp/CMakeFiles/ginkgo_omp.dir/base/batch_multi_vector_kernels.cpp.oESC[0m
cd /work2/00434/eijkhout/ginkgo/build-1.10.0-stampede3-gcc13.2-omp-omp/omp && /opt/apps/gcc/13.2.0/bin/g++ -DGKO_COMPILING_OMP -DGKO_DEVICE_NAMESPACE=omp -Dginkgo_omp_EXPORTS -I/work2/00434/eijkhout/ginkgo/ginkgo-1.10.0/omp -I/work2/00434/eijkhout/ginkgo/build-1.10.0-stampede3-gcc13.2-omp-omp/include -I/work2/00434/eijkhout/ginkgo/ginkgo-1.10.0/include -I/work2/00434/eijkhout/ginkgo/ginkgo-1.10.0 -O2 -g -DNDEBUG -fPIC -fopenmp -MD -MT omp/CMakeFiles/ginkgo_omp.dir/base/batch_multi_vector_kernels.cpp.o -MF CMakeFiles/ginkgo_omp.dir/base/batch_multi_vector_kernels.cpp.o.d -o CMakeFiles/ginkgo_omp.dir/base/batch_multi_vector_kernels.cpp.o -c /work2/00434/eijkhout/ginkgo/ginkgo-1.10.0/omp/base/batch_multi_vector_kernels.cpp

but apparently not during linking:

[ 31%] ESC[1mESC[32mLinking CXX executable abstract_factoryESC[0m
cd /work2/00434/eijkhout/ginkgo/build-1.10.0-stampede3-gcc13.2-omp-omp/core/test/base && /home1/apps/cmake/4.1.0/bin/cmake -E cmake_link_script CMakeFiles/core_test_base_abstract_factory.dir/link.txt --verbose=1
/opt/apps/gcc/13.2.0/bin/ld: ../../../lib/libginkgo_omp.so.1.10.0: undefined reference to `GOMP_single_start'
/opt/apps/gcc/13.2.0/bin/ld: ../../../lib/libginkgo_omp.so.1.10.0: undefined reference to `omp_destroy_lock'
/opt/apps/gcc/13.2.0/bin/ld: ../../../lib/libginkgo_omp.so.1.10.0: undefined reference to `omp_get_max_threads'
/opt/apps/gcc/13.2.0/bin/ld: ../../../lib/libginkgo_omp.so.1.10.0: undefined reference to `omp_get_num_threads'
/opt/apps/gcc/13.2.0/bin/ld: ../../../lib/libginkgo_omp.so.1.10.0: undefined reference to `omp_init_lock'
/opt/apps/gcc/13.2.0/bin/ld: ../../../lib/libginkgo_omp.so.1.10.0: undefined reference to `GOMP_barrier'
/opt/apps/gcc/13.2.0/bin/ld: ../../../lib/libginkgo_omp.so.1.10.0: undefined reference to `GOMP_parallel'
/opt/apps/gcc/13.2.0/bin/ld: ../../../lib/libginkgo_omp.so.1.10.0: undefined reference to `omp_unset_lock'
/opt/apps/gcc/13.2.0/bin/ld: ../../../lib/libginkgo_omp.so.1.10.0: undefined reference to `omp_set_lock'
/opt/apps/gcc/13.2.0/bin/ld: ../../../lib/libginkgo_omp.so.1.10.0: undefined reference to `omp_get_thread_num'

[yhmtsai]

Hi @VictorEijkhout Thanks for reporting.
Could you give more detail of your compiler setup and CMake Version?
The followings may help

• ldd lib/libginkgo_omp.so check whether it can find the libgomp.so
• check the file build/core/test/baseCMakeFiles/core_test_base_abstract_factory.dir/link.txt contains any libgomp.so

[VictorEijkhout]

1. Using gcc13 and cmake 4.1
2. I'm not seeing libgomp being requested:

[staff ginkgo:817] ldd /work2/00434/eijkhout/ginkgo/build-1.10.0-stampede3-gcc13.2-omp-omp/lib/libginkgo_omp.so.1.10.0
        linux-vdso.so.1 (0x00007ffd0fd35000)
        /opt/apps/xalt/xalt/lib64/libxalt_init.so (0x0000147072292000)
        libginkgo_cuda.so.1.10.0 => /work2/00434/eijkhout/ginkgo/build-1.10.0-stampede3-gcc13.2-omp-omp/lib/libginkgo_cuda.so.1.10.0 (0x00001470720a7000)
        libginkgo_hip.so.1.10.0 => /work2/00434/eijkhout/ginkgo/build-1.10.0-stampede3-gcc13.2-omp-omp/lib/libginkgo_hip.so.1.10.0 (0x0000147071ebe000)
        libginkgo_dpcpp.so.1.10.0 => /work2/00434/eijkhout/ginkgo/build-1.10.0-stampede3-gcc13.2-omp-omp/lib/libginkgo_dpcpp.so.1.10.0 (0x0000147071cd9000)
        libginkgo_device.so.1.10.0 => /work2/00434/eijkhout/ginkgo/build-1.10.0-stampede3-gcc13.2-omp-omp/lib/libginkgo_device.so.1.10.0 (0x0000147071cd3000)
        libstdc++.so.6 => /opt/apps/gcc/13.2.0/lib64/libstdc++.so.6 (0x0000147071a00000)
        libm.so.6 => /usr/lib64/libm.so.6 (0x0000147071925000)
        libgcc_s.so.1 => /opt/apps/gcc/13.2.0/lib64/libgcc_s.so.1 (0x0000147071cad000)
        libc.so.6 => /usr/lib64/libc.so.6 (0x0000147071600000)
        /lib64/ld-linux-x86-64.so.2 (0x000014707389a000)

3. Here is link.txt (there was a slash missing in your path):

[staff build-1.10.0-stampede3-gcc13.2-omp-omp:825] grep gomp !$
grep gomp core/test/base/CMakeFiles/core_test_base_abstract_factory.dir/link.txt
[staff build-1.10.0-stampede3-gcc13.2-omp-omp:826] cat !$
cat core/test/base/CMakeFiles/core_test_base_abstract_factory.dir/link.txt
/opt/apps/gcc/13.2.0/bin/g++ -O2 -g -DNDEBUG -Xlinker --enable-new-dtags -Xlinker -rpath -Xlinker /opt/intel/oneapi/mpi/2021.11/lib -Xlinker -rpath -Xlinker /opt/intel/oneapi/mpi/2021.11/lib -Xlinker --enable-new-dtags CMakeFiles/core_test_base_abstract_factory.dir/abstract_factory.cpp.o -o abstract_factory  -Wl,-rpath,/work2/00434/eijkhout/ginkgo/build-1.10.0-stampede3-gcc13.2-omp-omp/lib:/opt/intel/oneapi/mpi/2021.11/lib:/scratch/projects/compilers/intel24.0/oneapi/intelpython/python3.9/lib ../../../lib/libginkgo_gtest_main.a ../../../lib/libginkgo.so.1.10.0 ../../../lib/libginkgo_omp.so.1.10.0 ../../../lib/libginkgo_cuda.so.1.10.0 ../../../lib/libginkgo_reference.so.1.10.0 ../../../lib/libginkgo_hip.so.1.10.0 ../../../lib/libginkgo_dpcpp.so.1.10.0 ../../../lib/libginkgo_device.so.1.10.0 /opt/intel/oneapi/mpi/2021.11/lib/libmpicxx.so /opt/intel/oneapi/mpi/2021.11/lib/libmpi.so /usr/lib64/librt.a /usr/lib64/libpthread.a /usr/lib64/libdl.a /scratch/projects/compilers/intel24.0/oneapi/intelpython/python3.9/lib/libgtest.so.1.14.0

[upsj]

It seems like the link line for libginkgo_omp might be missing libgomp, can you maybe send us the CMakeCache.txt file and the output from a fresh CMake build with verbose output (make VERBOSE=1, though you might have that enabled already with CMAKE_VERBOSE_MAKEFILES=ON)? Then we can see the command invocations more clearly.

Alternatively, you could also configure with the Ninja generator and send us the output of ninja -t commands core_test_base_abstract_factory, that gives us all build commands directly without building the project.

[yhmtsai]

Hi @VictorEijkhout
I guess these issues are related to CMake 4.1.0 with CLICOLOR_FORCE=1
Could you check whether you have set this variable?
Related Issue in CMake https://gitlab.kitware.com/cmake/cmake/-/issues/27137, https://gitlab.kitware.com/cmake/cmake/-/issues/27155.
It is fixed in CMake 4.1.1 (https://gitlab.kitware.com/cmake/cmake/-/merge_requests/11080 fixed it)
If you set it, could you try to compile ginkgo from scratch after unsetting CLICOLOR_FORCE?
upgrading CMake to 4.1.1 should also solve it