Cmake output saying 'Configuring kernel module without CUDA'

[babouFomb]

Hi;

I have a Jetson Xavier AGX kit board and I plugged into the M.2 key M an NVMe SSD. Now, I'm trying to install your libnm on my Xavier and I show the following message in CMake output:

-- Found CUDA: /usr/local/cuda-10.0 (found suitable version "10.0", minimum required is "8.0")
-- Using NVIDIA driver found in
-- Configuring kernel module without CUDA
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ganapathi/Downloads/ssd-gpu-dma-master/build

How can I force Cmake to build with CUDA?

Thank

[enfiskutensykkel]

Hi,

It's strange that it is that it is outputting that it found the Nvidia driver, but the string appears empty. Regardless, it is possible to override the location using -DNVIDIA=<path to driver source>, usually it can be found in /usr/src/nvidia-<driver version>. Make sure to run make in the Nvidia driver directory first.

[babouFomb]

But I can't found the path to driver source. I my /usr/src/ directory there is no nvidia-driver--.

ganapathi@ganapathi-desktop:/$ ls /usr/src/
cudnn_samples_v7 linux-headers-4.9.140-tegra-linux_x86_64 linux-headers-4.9.140-tegra-ubuntu18.04_aarch64 nvidia tensorrt
ganapathi@ganapathi-desktop:/$ ls /usr/src/linux-headers-4.9.140-tegra-ubuntu18.04_aarch64/
kernel-4.9 nvgpu nvidia
ganapathi@ganapathi-desktop:/$ ls /usr/src/linux-headers-4.9.140-tegra-ubuntu18.04_aarch64/kernel-4.9/
arch block certs crypto drivers firmware fs include init ipc Kbuild Kconfig kernel lib Makefile mm Module.symvers net samples scripts security sound tools usr virt

I try to find it with the find iname nvidia-driver but I got juste

ganapathi@ganapathi-desktop:/$ sudo find -iname nvidia-driver
find: ‘./run/user/1000/gvfs’: Permission denied
find: ‘./run/user/120/gvfs’: Permission denied

[enfiskutensykkel]

I'm not really familiar with the Xavier, but you can try locating the directory manually by running find /usr/src -name "nv-p2p.h". However, it's possible that the Nvidia driver symbols are built into the kernel source.

[enfiskutensykkel]

You can attempt using the either -DNVIDIA=/usr/src/linux-headers-4.9.140-tegra-ubuntu18.04_aarch64/nvgpu/include/linux/ or -DNVIDIA=/usr/src/linux-headers-4.9.140-tegra-linux_x86_64/nvgpu/include/linux/

Hope it works!

[babouFomb]

Hi,
I used the -DNVIDIA=/usr/src/linux-headers-4.9.140-tegra-ubuntu18.04_aarch64/nvgpu/include/linux/ flag. Now, CMake find the path to the driver, but it still configuring kernel module without CUDA

-- Found CUDA: /usr/local/cuda-10.0 (found suitable version "10.0", minimum required is "8.0")
-- Using NVIDIA driver found in /usr/src/linux-headers-4.9.140-tegra-ubuntu18.04_aarch64/nvgpu/include
-- Configuring kernel module without CUDA
-- Configuring done
-- Generating done
-- Build files have been written to: /home/ganapathi/Downloads/ssd-gpu-dma-master/build

[enfiskutensykkel]

Hi,

Yes, it's looking for the Module.symvers but on the Tegra the Nvidia driver seems to be compiled in to the kernel.

You can either try modifying this line and remove the check for Module.symvers
https://github.com/enfiskutensykkel/ssd-gpu-dma/blob/master/CMakeLists.txt#L144
if (CUDA_FOUND AND NOT no_cuda)

If that doesn't work, you could try modifying this line and add -D_CUDA:
https://github.com/enfiskutensykkel/ssd-gpu-dma/blob/master/CMakeLists.txt#L149
set (module_ccflags "-D_CUDA -I${libnvm_root}")

The third option is perhaps modifying the generated Makefile for the kernel module after running CMake.

[enfiskutensykkel]

It may actually be easier to just add -D_CUDA to this line in Makefile.in:
https://github.com/enfiskutensykkel/ssd-gpu-dma/blob/master/module/Makefile.in#L6

You may need to include the header file location, so the line becomes:
ccflags-y += @module_ccflags@ -D_CUDA -I/usr/src/linux-headers-4.9.140-tegra-ubuntu18.04_aarch64/nvgpu/include/linux

[babouFomb]

I think the problem comes from finding Nvidia driver symbols (lines 66 - 68). I tried to locate the Module.symvers with find /usr/src -name "Module.symvers" and I obtain this output

/usr/src/linux-headers-4.9.140-tegra-linux_x86_64/kernel-4.9/Module.symvers
/usr/src/linux-headers-4.9.140-tegra-ubuntu18.04_aarch64/kernel-4.9/Module.symvers

So I checked in CMake, and remove the fird condition in the line 144 if (CUDA_FOUND AND NOT no_cuda AND EXISTS "${driver_dir}/Module.symvers") and then replace the line 146 with set (module_symbols "/usr/src/linux-headers-4.9.140-tegra-ubuntu18.04_aarch64/kernel-4.9/Module.symvers"). I also replaced the line 6 in Makefile.in by ccflags-y += @module_ccflags@ -D_CUDA -I/usr/src/linux-headers-4.9.140-tegra-ubuntu18.04_aarch64/nvgpu/include/linux. Now, CMake output is

-- Found CUDA: /usr/local/cuda-10.0 (found suitable version "10.0", minimum required is "8.0")
-- Using NVIDIA driver found in /usr/src/linux-headers-4.9.140-tegra-ubuntu18.04_aarch64/nvgpu/include
-- Configuring kernel module with CUDA
-- Configuring done
-- Generating done

The commands make libvnm and make examples run successfully. Since I will not use SISCI SmartIO, I go in module directory (inside my build) dir and run make. I obtained several compilation errors (seen in the attached files)
log.txt

[enfiskutensykkel]

It appears that the symbols are wildly different on the Xavier/Tegra kernel than for x86.
https://docs.nvidia.com/cuda/gpudirect-rdma/index.html#kernel-api
I need to get my hands on a Xavier in order to make those fixes.

Regardless, my understanding is that the main system memory is shared by the on-board GPU on all of the Tegra SoCs, so it might be the case that those calls aren't really necessary to begin with. I need to do some more investigations into that.

[babouFomb]

I make some modification in the map.c file in order to port it to Xavier:

Adding struct device* dev; in struct map in map.h
Removing the arguments map->pdev, gd->pages from nvidia_p2p_dma_unmap_pages (line 257 in map.c)
Removing the arguments 0, 0, map->vaddr from nvidia_p2p_put_pages (line 262 in map.c)
Adding enum dma_data_direction direction to map_gpu_memory formal arguments (line 276 in map.c)
Removing the arguments map->vaddr, GPU_PAGE_SIZE * map->n_addrs from nvidia_p2p_get_pages (line 296 in map.c)
Replacing the 1st argument (struct pci_dev) of nvidia_p2p_dma_map_pages by dev of type struct device* and adding a new argument of type enum dma_data_direction direction (line 303 in map.c)
In line 346 in map.c, I added 0 since I previously added the supplementary enum dma_data_direction direction argument to map_gpu_memory. In the file linux/dma-mapping.h, the value 0 corresponds to DMA_BIDIRECTIONAL (I am not sure choosing 0 is a good idea)

After these modifications, the module has been successfully compiled without any error. However, when I tried the identify example, I get the following output :

ganapathi@ganapathi-desktop:~/Downloads/ssd-gpu-dma-master/build/bin$ lspci
0000:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad0 (rev a1)
0000:01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981
0001:00:00.0 PCI bridge: NVIDIA Corporation Device 1ad2 (rev a1)
0001:01:00.0 SATA controller: Marvell Technology Group Ltd. Device 9171 (rev 13)

ganapathi@ganapathi-desktop:~/Downloads/ssd-gpu-dma-master/build$ sudo ./bin/nvm-identify-userspace --ctrl=01:00.0
[sudo] password for ganapathi:
Resetting controller and setting up admin queues...
Failed to get number of queues
Goodbye!

[enfiskutensykkel]

After these modifications, the module has been successfully compiled without any error.

Great!

However, when I tried the identify example, I get the following output :

If you have loaded the module, then you should invoke the ./bin/nvm-identify example, not the userspace variant. The ctrl argument should be the character device in /dev (I believe it's likely to be /dev/libnvm0, you can also check the output in system log by running dmesg).

Apologies for the documentation, it isn't really up to date.

That being said, it seems to me that the DMA is not working properly.
I'm not sure if it is possible to disable the IOMMU on Xavier, but it might be the case that it is always on. I don't think the module sets up DMA mappings properly when the IOMMU is enabled. You can look for DMA faults in the system log (dmesg).

Additionally, I also believe that some of the Tegras are not cache coherent. If Xavier isn't as well, then you might need to add code to flush the cache in the queue functions.

[babouFomb]

I have loaded the module sudo insmod libnvm.ko and then verify that is has been loaded properly :

ganapathi@ganapathi-desktop:~/Downloads/ssd-gpu-dma-master/build/module$ lsmod
Module Size Used by
libnvm 12856 0

I run the identify example with sudo ./nvm-identify --ctrl=/dev/nvme0n1, but I got nothing as output. :

ganapathi@ganapathi-desktop:/Downloads/ssd-gpu-dma-master/build/bin$ sudo ./nvm-identify --ctrl=/dev/nvme0n1
ganapathi@ganapathi-desktop:/Downloads/ssd-gpu-dma-master/build/bin$ sudo ./nvm-identify --ctrl=/dev/nvme0n1

I also verified if the IOMMU is enabled with cat /proc/cmdline | grep iommu and the output is empty so I suppose the IOMMU is disabled.

I looked for DMA faults with dmesg I seen this :

ganapathi@ganapathi-desktop:~/Downloads/ssd-gpu-dma-master/build/bin$ dmesg | grep dma
[ 1.023726] iommu: Adding device 2600000.dma to group 56
[ 1.302411] tegra-carveouts tegra-carveouts: vpr :dma coherent mem declare 0x00000000f0000000,134217728
[ 1.304883] tegra-gpcdma 2600000.dma: GPC DMA driver register 31 channels
[ 1.965973] tegradccommon 15200000.dc_common: dma mapping done
[ 1.978965] tegra-adma 2930000.adma: Tegra210 ADMA driver registered 16 channels
[ 5.752450] misc nvmap: cvsram :dma coherent mem declare 0x0000000050000000,4194304

[enfiskutensykkel]

--ctrl=/dev/nvme0n1

This is not the character device created by the nvme module, this is the block device created by the built-in Linux NVMe driver. You need to unbind the driver for the NVMe and then reload the libnvm driver.