Build with compiler enabled through Environment Modules

[randomparity]

Attempting to build DPDK on a system with gcc 11 provided by an Environment Module, specifically the IBM Advanced Toolkit AT 15.0 on a POWER system (not cross compiled). Linking stage fails like this:

cc  -o app/dpdk-test-gpudev app/dpdk-test-gpudev.p/test-gpudev_main.c.o -Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -Wl,--whole-archive -Wl,--start-group <... project libraries skipped ...> -Wl,--no-whole-archive -Wl,--no-as-needed -pthread -lm -ldl -lnuma -lfdt -Wl,-rpath,/lib/../lib64:XXXXXXXXXXXXXXXXXX -Wl,-rpath-link,/lib/../lib64 -Wl,--export-dynamic /usr/lib64/libpcap.so /usr/lib64/libelf.so /lib/../lib64/libmlx5.so /lib/../lib64/libibverbs.so /usr/lib64/libcrypto.so /usr/lib64/libz.so /lib/../lib64/libmlx4.so -lrt -Wl,--end-group
/opt/at15.0/lib/gcc/powerpc64le-linux-gnu/11.2.1/../../../../powerpc64le-linux-gnu/bin/ld: /lib/../lib64/libpthread.so.0: undefined reference to `__libc_dlopen_mode@GLIBC_PRIVATE'
/opt/at15.0/lib/gcc/powerpc64le-linux-gnu/11.2.1/../../../../powerpc64le-linux-gnu/bin/ld: /lib/../lib64/libpthread.so.0: undefined reference to `__nanosleep_nocancel@GLIBC_PRIVATE'
... <additional errors skipped> ...

Problem appears to be related to the fact that libpthread and libdl have been merged into glibc 2.34 (see Red Hat Article but meson is looking in the wrong locations for the standard library:

$ gcc --print-file-name=libc.so
/usr/lib/gcc/ppc64le-redhat-linux/8/../../../../lib64/libc.so
$ module load at15.0
$ gcc --print-file-name=libc.so
/opt/at15.0/lib/../lib64/libc.so

Is there an option that would allow meson to find the correct libdir when running with a compiler enabled through an environment module?

[QuLogic]

This is a bug in DPDK; they do the linking completely wrong:

# if link_lib is empty, do not add it to project properties
if link_lib != ''
    add_project_link_arguments('-l' + link_lib, language: 'c')
    dpdk_extra_ldflags += '-l' + link_lib
endif

and similarly for lm, lpthread, etc.

They should be using dependencies, e.g., https://mesonbuild.com/Dependencies.html#dl-libdl or for older versions, using compiler.find_library('dl', required : false).

[eli-schwartz]

What's interesting is that this shouldn't really matter -- if you do in fact do cc.find_library('dl', required: false) then meson special cases this as UNIXY_COMPILER_INTERNAL_LIBS then returns the -l version rather than /path/to/libdl.so.

Either way, the compiler is supposed to special case this name and search in the same directory as its configured libc, but apparently cc -ldl misfired.

[eli-schwartz]

cc -o outfile ... -pthread -lm -ldl -lnuma -lfdt ...
/.../ld: /lib/../lib64/libpthread.so.0: undefined reference to `__libc_dlopen_mode@GLIBC_PRIVATE'
/.../ld: /lib/../lib64/libpthread.so.0: undefined reference to `__nanosleep_nocancel@GLIBC_PRIVATE'
... <additional errors skipped> ...

These names are supposed to be specially known to the compiler which will search for them next to libc.

$ gcc --print-file-name=libc.so
/usr/lib/gcc/ppc64le-redhat-linux/8/../../../../lib64/libc.so
$ module load at15.0
$ gcc --print-file-name=libc.so
/opt/at15.0/lib/../lib64/libc.so

This really says nothing. You're getting errors detecting -ldl, not -lc.

Also, glibc is supposed to install compatibility static libraries (libdl.a) that contain no objects, to ensure that these still resolve to files on disk if needed.

All in all, it initially seems to me that something about this compiler may be fishy. Incidentally, why does it have its own libc anyway?

[drchristensen]

Spoke with some internal toolset developers and they suggest the problem is -Wl,-rpath-link,/lib/../lib64, which is forcing the system libraries to be used, not the libraries in /opt/at15.0/lib64. I spent some time implementing dependencies() and compiler.find_library() changes recommended by @QuLogic but the error persists. For example, existing code:

if is_linux
    link_lib = 'dl'
else
    link_lib = ''
endif

# if link_lib is empty, do not add it to project properties
if link_lib != ''
    add_project_link_arguments('-l' + link_lib, language: 'c')
    dpdk_extra_ldflags += '-l' + link_lib
endif

was changed to

 if is_linux
    if meson.version().version_compare('<=0.62.0')
        dl_dep = cc.find_library('dl', required : false)
    else
        dl_dep = dependency('dl')
    endif
    if dl_dep.found()
        base_dep_objs += dl_dep
    endif
endif
...
executable(... dependencies: base_dep_objs ...)

Even with the changes, the suspect linker option -Wl,-rpath-link,/lib/../lib64 still lives in the ninja.build file and the link stage fails as previously described. Looking through other meson.build files in the project I can't find any project code that sets this command line option. I can see other linker options explicitly set with add_project_link_arguments() or default_ldflags += ['-Wl,--export-dynamic'] both before and after the suspect linker option on the LINK_ARGS line, but the rpath and rpath-link settings seem to come from meson itself, not any DPDK project files. How are they selected and how can I influence them?

Also, any idea where the XXXX may be coming from in the link option -Wl,-rpath,/lib/../lib64:XXXXXXXXXXXXXXXXXX -Wl,-rpath-link,/lib/../lib64? When I build with the system c compiler (gcc 8.5), the option is -Wl,-rpath,XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX and there's no rpath-link option.

Regarding why their own libc, the Advanced Toolchain (AT) website says: GNU C library (glibc), individually optimized for supported POWER processors, which I take to mean that the library is built with gcc11 to take advantage of POWER[8/9/10] specific optimizations enabled in that version of the compiler.

[eli-schwartz]

Hmm. dpdk does set

install_rpath: join_paths(get_option('prefix'), driver_install_path),

[drchristensen]

Commented both instance out, same error:

git diff app/meson.build
diff --git a/app/meson.build b/app/meson.build
index 93d8c15032..4a7d2e7f07 100644
--- a/app/meson.build
+++ b/app/meson.build
@@ -70,9 +70,9 @@ foreach app:apps
             c_args: cflags,
             link_args: ldflags,
             link_whole: link_libs,
-            dependencies: ext_deps + dep_objs,
+            dependencies: base_dep_objs + ext_deps + dep_objs,
             include_directories: includes,
-            install_rpath: join_paths(get_option('prefix'), driver_install_path),
+            # install_rpath: join_paths(get_option('prefix'), driver_install_path),
             install: true)
 endforeach

$ git diff app/test/meson.build
diff --git a/app/test/meson.build b/app/test/meson.build
index 5fc1dd1b7b..91f0e4af6b 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -453,10 +453,10 @@ endif
 dpdk_test = executable('dpdk-test',
         test_sources,
         link_whole: link_libs,
-        dependencies: test_dep_objs + ext_deps,
+        dependencies: base_dep_objs + test_dep_objs + ext_deps,
         c_args: cflags,
-        install_rpath: join_paths(get_option('prefix'),
-             driver_install_path),
+        # install_rpath: join_paths(get_option('prefix'),
+        #     driver_install_path),
         install: true)

 has_hugepage = run_command(py3, 'has_hugepage.py', check: true).stdout().strip() != '0'

[drchristensen]

Discovered that if I manually edit the ninja.build file with the following commands then the build completes successfully:

sed --in-place=.BAK1 's|-Wl,-rpath,/lib/../lib64||' build.ninja
sed --in-place=.BAK2 's|-Wl,-rpath-link,/lib/../lib64||' build.ninja

This confirms the advice I received from some local gcc experts on the source of the problem but doesn't help me understand why it's an issue with Meson. Is this behavior actually a Meson feature, mentioned in the 0.55.0 release notes that says

On Linux-like systems, Meson adds rpath entries to allow running apps in the
build tree, and then removes those build-time-only rpath entries when
installing.

Is there any possibility of modifying this behavior to work in this scenario?