cuTENSOR: Valgrind report segFault when runing contraction_jit due to invalid memory access

[Hanfei147]

Hi, When running the contraction_jit sample with Valgrind (valgrind --tool=memcheck), the program crashes with a segmentation fault during cutensorWriteKernelCacheToFile. The error indicates an invalid read in memmove.

Environment:

• OS: Ubuntu 20.04
• CUDA: 12.8
• CUDA driver: 570.172.08
• cuTensor: 2.1.0
• Valgrind: 3.15.0

Repro Steps:

1. Build the contraction_jit sample
2. Run once to ensure kernelCache.bin is generated in the directory
3. Then run with valgrind:

valgrind --tool=memcheck --track-origins=yes ./contraction_jit

Valgrind Output:

Total memory: 0.38 GiB
Kernel cache found and read successfully.
cuTENSOR    :    865 GFLOPs/s
cuTENSOR JIT:    866 GFLOPs/s
Speedup: 1.0x
JIT Compilation time: 0.0 seconds (Kernel cache file was read successfully; Compilation was not required)
Kernel cache written to file. Will be read in next execution.
Successful completion
root@1c52d552ef27:~/CUDALibrarySamples/cuTENSOR# valgrind --tool=memcheck --track-origins=yes ./contraction_jit
==1035== Memcheck, a memory error detector
==1035== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==1035== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==1035== Command: ./contraction_jit
==1035==
==1035== Warning: set address range perms: large range [0x484a000, 0x1dd2a000) (defined)
==1035== Warning: set address range perms: large range [0x1e159000, 0x5044b000) (defined)
Total memory: 0.38 GiB
==1035== Warning: noted but unhandled ioctl 0x30000001 with no size/direction hints.
==1035==    This could cause spurious value errors to appear.
==1035==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==1035== Warning: noted but unhandled ioctl 0x4b with no size/direction hints.
==1035==    This could cause spurious value errors to appear.
==1035==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==1035== Warning: noted but unhandled ioctl 0x27 with no size/direction hints.
==1035==    This could cause spurious value errors to appear.
==1035==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==1035== Warning: noted but unhandled ioctl 0x25 with no size/direction hints.
==1035==    This could cause spurious value errors to appear.
==1035==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==1035== Warning: noted but unhandled ioctl 0x17 with no size/direction hints.
==1035==    This could cause spurious value errors to appear.
==1035==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==1035== Warning: set address range perms: large range [0x200000000, 0xa00200000) (noaccess)
==1035== Warning: noted but unhandled ioctl 0x19 with no size/direction hints.
==1035==    This could cause spurious value errors to appear.
==1035==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==1035== Warning: noted but unhandled ioctl 0x49 with no size/direction hints.
==1035==    This could cause spurious value errors to appear.
==1035==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==1035== Warning: noted but unhandled ioctl 0x21 with no size/direction hints.
==1035==    This could cause spurious value errors to appear.
==1035==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==1035== Warning: noted but unhandled ioctl 0x1b with no size/direction hints.
==1035==    This could cause spurious value errors to appear.
==1035==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==1035== Warning: noted but unhandled ioctl 0x44 with no size/direction hints.
==1035==    This could cause spurious value errors to appear.
==1035==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==1035== Warning: set address range perms: large range [0x7a400000, 0x8c3ff000) (noaccess)
Kernel cache found and read successfully.
cuTENSOR    :    812 GFLOPs/s
cuTENSOR JIT:    812 GFLOPs/s
Speedup: 1.0x
JIT Compilation time: 0.0 seconds (Kernel cache file was read successfully; Compilation was not required)
==1035== Invalid read of size 1
==1035==    at 0x4842A39: memmove (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==1035==    by 0x4E382A1: ??? (in /opt/nvidia/hpc_sdk/Linux_x86_64/25.3/math_libs/12.8/targets/x86_64-linux/lib/libcutensor.so.2.1.0)
==1035==    by 0x4E37419: ??? (in /opt/nvidia/hpc_sdk/Linux_x86_64/25.3/math_libs/12.8/targets/x86_64-linux/lib/libcutensor.so.2.1.0)
==1035==    by 0x4DC1E0D: cutensorWriteKernelCacheToFile (in /opt/nvidia/hpc_sdk/Linux_x86_64/25.3/math_libs/12.8/targets/x86_64-linux/lib/libcutensor.so.2.1.0)
==1035==    by 0x112994: main (contraction_jit.cu:482)
==1035==  Address 0xc90385cc is not stack'd, malloc'd or (recently) free'd
==1035==
==1035==
==1035== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==1035==  Access not within mapped region at address 0xC90385CC
==1035==    at 0x4842A39: memmove (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==1035==    by 0x4E382A1: ??? (in /opt/nvidia/hpc_sdk/Linux_x86_64/25.3/math_libs/12.8/targets/x86_64-linux/lib/libcutensor.so.2.1.0)
==1035==    by 0x4E37419: ??? (in /opt/nvidia/hpc_sdk/Linux_x86_64/25.3/math_libs/12.8/targets/x86_64-linux/lib/libcutensor.so.2.1.0)
==1035==    by 0x4DC1E0D: cutensorWriteKernelCacheToFile (in /opt/nvidia/hpc_sdk/Linux_x86_64/25.3/math_libs/12.8/targets/x86_64-linux/lib/libcutensor.so.2.1.0)
==1035==    by 0x112994: main (contraction_jit.cu:482)
==1035==  If you believe this happened as a result of a stack
==1035==  overflow in your program's main thread (unlikely but
==1035==  possible), you can try to increase the size of the
==1035==  main thread stack using the --main-stacksize= flag.
==1035==  The main thread stack size used in this run was 8388608.
==1035==
==1035== HEAP SUMMARY:
==1035==     in use at exit: 737,323,325 bytes in 307,997 blocks
==1035==   total heap usage: 450,372 allocs, 142,375 frees, 900,451,450 bytes allocated
==1035==
==1035== LEAK SUMMARY:
==1035==    definitely lost: 0 bytes in 0 blocks
==1035==    indirectly lost: 0 bytes in 0 blocks
==1035==      possibly lost: 143,632 bytes in 1,417 blocks
==1035==    still reachable: 737,179,693 bytes in 306,580 blocks
==1035==         suppressed: 0 bytes in 0 blocks
==1035== Rerun with --leak-check=full to see details of leaked memory
==1035==
==1035== For lists of detected and suppressed errors, rerun with: -s
==1035== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 4 from 4)
Segmentation fault

[Hanfei147]

One more thing, it run normally without valgrind. I want to know whether it is a problem with the cutensor library or my own configuration.

[ChrisPsa]

Hi Hanfei147,

would it be possible to update to the latest version of cuTENSOR (2.3.0)?
You might also need to update your CUDA driver and CUDA accordingly.

[Hanfei147]

Hi Chris, thanks for your reply

I have upgraded cutensor to version 2.3.1, CUDA to 13.0 and CUDA driver to 580.65.06, but a segmentation fault still occurs at the relevant location when running under Valgrind. The Valgrind output log is as follows:

==3206== Memcheck, a memory error detector
==3206== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==3206== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==3206== Command: ./contraction_jit
==3206==
==3206== Warning: set address range perms: large range [0x484a000, 0x1535d000) (defined)
==3206== Warning: set address range perms: large range [0x15a06000, 0x3b3e6000) (defined)
Total memory: 0.38 GiB
==3206== Warning: noted but unhandled ioctl 0x30000001 with no size/direction hints.
==3206==    This could cause spurious value errors to appear.
==3206==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==3206== Warning: noted but unhandled ioctl 0x4b with no size/direction hints.
==3206==    This could cause spurious value errors to appear.
==3206==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==3206== Warning: noted but unhandled ioctl 0x27 with no size/direction hints.
==3206==    This could cause spurious value errors to appear.
==3206==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==3206== Warning: noted but unhandled ioctl 0x25 with no size/direction hints.
==3206==    This could cause spurious value errors to appear.
==3206==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==3206== Warning: noted but unhandled ioctl 0x17 with no size/direction hints.
==3206==    This could cause spurious value errors to appear.
==3206==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==3206== Warning: set address range perms: large range [0x200000000, 0xa00200000) (noaccess)
==3206== Warning: noted but unhandled ioctl 0x19 with no size/direction hints.
==3206==    This could cause spurious value errors to appear.
==3206==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==3206== Warning: noted but unhandled ioctl 0x49 with no size/direction hints.
==3206==    This could cause spurious value errors to appear.
==3206==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==3206== Warning: noted but unhandled ioctl 0x21 with no size/direction hints.
==3206==    This could cause spurious value errors to appear.
==3206==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==3206== Warning: noted but unhandled ioctl 0x1b with no size/direction hints.
==3206==    This could cause spurious value errors to appear.
==3206==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==3206== Warning: noted but unhandled ioctl 0x44 with no size/direction hints.
==3206==    This could cause spurious value errors to appear.
==3206==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on writing a proper wrapper.
==3206== Warning: set address range perms: large range [0x68000000, 0x79fff000) (noaccess)
Kernel cache found and read successfully.
cuTENSOR    :    813 GFLOPs/s
cuTENSOR JIT:   7357 GFLOPs/s
Speedup: 9.1x
JIT Compilation time: 0.6 seconds (Kernel cache file was read successfully; Compilation was not required)
==3206== Invalid read of size 1
==3206==    at 0x4842A39: memmove (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==3206==    by 0x4E0B221: ??? (in /root/libcutensor-linux-x86_64-2.3.1.0_cuda13-archive/lib/libcutensor.so.2.3.1)
==3206==    by 0x4E0A559: ??? (in /root/libcutensor-linux-x86_64-2.3.1.0_cuda13-archive/lib/libcutensor.so.2.3.1)
==3206==    by 0x4D88B0D: cutensorWriteKernelCacheToFile (in /root/libcutensor-linux-x86_64-2.3.1.0_cuda13-archive/lib/libcutensor.so.2.3.1)
==3206==    by 0x10C464: main (in /root/CUDALibrarySamples/cuTENSOR/contraction_jit)
==3206==  Address 0xb557606c is not stack'd, malloc'd or (recently) free'd
==3206==
==3206==
==3206== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==3206==  Access not within mapped region at address 0xB557606C
==3206==    at 0x4842A39: memmove (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==3206==    by 0x4E0B221: ??? (in /root/libcutensor-linux-x86_64-2.3.1.0_cuda13-archive/lib/libcutensor.so.2.3.1)
==3206==    by 0x4E0A559: ??? (in /root/libcutensor-linux-x86_64-2.3.1.0_cuda13-archive/lib/libcutensor.so.2.3.1)
==3206==    by 0x4D88B0D: cutensorWriteKernelCacheToFile (in /root/libcutensor-linux-x86_64-2.3.1.0_cuda13-archive/lib/libcutensor.so.2.3.1)
==3206==    by 0x10C464: main (in /root/CUDALibrarySamples/cuTENSOR/contraction_jit)
==3206==  If you believe this happened as a result of a stack
==3206==  overflow in your program's main thread (unlikely but
==3206==  possible), you can try to increase the size of the
==3206==  main thread stack using the --main-stacksize= flag.
==3206==  The main thread stack size used in this run was 8388608.
==3206==
==3206== HEAP SUMMARY:
==3206==     in use at exit: 747,895,256 bytes in 215,099 blocks
==3206==   total heap usage: 291,464 allocs, 76,365 frees, 956,317,892 bytes allocated
==3206==
==3206== LEAK SUMMARY:
==3206==    definitely lost: 0 bytes in 0 blocks
==3206==    indirectly lost: 0 bytes in 0 blocks
==3206==      possibly lost: 131,760 bytes in 1,852 blocks
==3206==    still reachable: 747,763,496 bytes in 213,247 blocks
==3206==         suppressed: 0 bytes in 0 blocks
==3206== Rerun with --leak-check=full to see details of leaked memory
==3206==
==3206== For lists of detected and suppressed errors, rerun with: -s
==3206== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 4 from 4)
Segmentation fault

[ChrisPsa]

Hi Hanfei147,

I have a few more questions:

1. what GPU are you running on?
2. You mentioned that this issue does not occur when running the example outside of valgrind. Is this still the case?
3. Have you tried removing any leftover kernelCache.bin file?

Thank you,
Christos

[Hanfei147]

Hi Hanfei147,

I have a few more questions:

1. what GPU are you running on?
2. You mentioned that this issue does not occur when running the example outside of valgrind. Is this still the case?
3. Have you tried removing any leftover kernelCache.bin file?

Thank you, Christos

Hi Christos,

1. What GPU are you running on?
   I am running the test on an NVIDIA A10 GPU (driver version: 580.65.06, CUDA version: 13.0, cuTENSOR 2.3.1).
2. Does the issue persist when not using Valgrind?
   This issue only occur when running with Valgrind.
3. Have you tried removing any leftover kernelCache.bin file?
   Yes, here's the observed behavior:
   -- Without Valgrind: The example runs stably across multiple executions (no segfaults).
   -- With Valgrind: First run, works normally (no segFault, generates a new kernelCache.bin). Second run, Segfaults occur in cutensorWriteKernelCacheToFile.

The issue is reproducible and I think it directly tied to the reuse of the cached kernel file under Valgrind. Could you verify whether the functions cutensorReadKernelCacheFromFile and cutensorWriteKernelCacheToFile perform read/write operations on a block of memory that has already been freed when kernelCache.bin exists.