Issue with SUP basecalling using RTX3080 10GB

[awjga]

New issue checks

• I have read the Dorado Documentation.
• I did not find an existing issue.

Dorado version

1.1.1

Dorado subcommand

Basecaller

The issue

dorado basecaller sup input.pod5 > basecalled.bam

Output:
[2025-09-04 15:12:33.631] [info] > Creating basecall pipeline
[2025-09-04 15:12:34.581] [info] Using CUDA devices:
[2025-09-04 15:12:34.581] [info] cuda:0 - NVIDIA GeForce RTX 3080
[2025-09-04 15:12:35.444] [info] Calculating optimized batch size for GPU "NVIDIA GeForce RTX 3080" and model dna_r10.4.1_e8.2_400bps_sup@v5.2.0. Full benchmarking will run for this device, which may take some time.
[2025-09-04 15:12:36.267] [info] cuda:0 using chunk size 12288, batch size 96
[2025-09-04 15:12:36.475] [info] cuda:0 using chunk size 6144, batch size 192
[2025-09-04 15:14:27.768] [warning] Caught Torch error 'CUDA error: unspecified launch failure
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
', clearing CUDA cache and retrying.
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: unspecified launch failure
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /builds/machine-learning/torch-builds/pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0xb0 (0x7f220d205390 in /mnt/SSD1/dorado-1.1.1-linux-x64/bin/../lib/libdorado_torch_lib.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 0xfa (0x7f2203da5896 in /mnt/SSD1/dorado-1.1.1-linux-x64/bin/../lib/libdorado_torch_lib.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3cc (0x7f220d1a639c in /mnt/SSD1/dorado-1.1.1-linux-x64/bin/../lib/libdorado_torch_lib.so)
frame #3: + 0xb245e1d (0x7f220d181e1d in /mnt/SSD1/dorado-1.1.1-linux-x64/bin/../lib/libdorado_torch_lib.so)
frame #4: + 0xb24677e (0x7f220d18277e in /mnt/SSD1/dorado-1.1.1-linux-x64/bin/../lib/libdorado_torch_lib.so)
frame #5: + 0xb259f98 (0x7f220d195f98 in /mnt/SSD1/dorado-1.1.1-linux-x64/bin/../lib/libdorado_torch_lib.so)
frame #6: /mnt/SSD1/dorado-1.1.1-linux-x64/bin/dorado() [0x54d7d4]
frame #7: + 0xc2b23 (0x7f2201bf7b23 in /mnt/SSD1/dorado-1.1.1-linux-x64/bin/../lib/libstdc++.so.6)
frame #8: + 0x8609 (0x7f2201ed2609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #9: clone + 0x43 (0x7f22018f6353 in /lib/x86_64-linux-gnu/libc.so.6)

System specifications

Operating system:Ubuntu 20.04.6 LTS
CPU: AMD Ryzen 9 3950X 16-Core Processor
GPU: RTX 3080 10 gb
SSD

Hi,

I am facing an issue when try to use sup to basecall my reads, after starting the command for a few minutes the GPU would crash and I would need to reboot the system to to use the GPU.
I have also tried to lower the batch size to -b 12 and the same error occurs.
When I tried using HAC for basecalling the same pod5 files, there was no issue with the same system.

Thank you

[malton-ont]

Hi @awjga,

Can you please post the output of nvidia-smi and ldd /mnt/SSD1/dorado-1.1.1-linux-x64/bin/dorado?

[awjga]

Hi @malton-ont,

These are the output:

 ldd /mnt/SSD1/dorado-1.1.1-linux-x64/bin/dorado
        linux-vdso.so.1 (0x00007fffeb1b4000)
        libdorado_torch_lib.so => /mnt/SSD1/dorado-1.1.1-linux-x64/bin/../lib/libdorado_torch_lib.so (0x00007f8bea4b3000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f8bea487000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f8bea46b000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f8bea465000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8bea442000)
        libstdc++.so.6 => /mnt/SSD1/dorado-1.1.1-linux-x64/bin/../lib/libstdc++.so.6 (0x00007f8bea0ad000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8be9f5c000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f8be9f41000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8be9d4f000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f8c65b2e000)
        libdorado_cudnn_lib.so => /mnt/SSD1/dorado-1.1.1-linux-x64/bin/../lib/../lib/libdorado_cudnn_lib.so (0x00007f8ba3468000)

nvidia-smi
Tue Sep  9 13:49:27 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.172.08             Driver Version: 570.172.08     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3080        Off |   00000000:09:00.0 Off |                  N/A |
|  0%   30C    P8              5W /  320W |     313MiB /  10240MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            1694      C   ...do/bin/dorado_basecall_server        214MiB |
|    0   N/A  N/A            2038      G   /usr/lib/xorg/Xorg                       56MiB |
|    0   N/A  N/A            2091      G   /usr/bin/gnome-shell                      8MiB |
+-----------------------------------------------------------------------------------------+

[malton-ont]

Thanks, that all looks in order. Are you able to run the CUDA samples tests successfully?

[malton-ont]

Can you also produce the logs with the -vv flag set? This will provide a bit more information during the batchsize benchmarking process.

[awjga]

Hi @malton-ont,

Thank you for the reply.

I am unable to build CUDA samples tests as I will need to install a newer version of cmake.
I tired running the basecalling again for both hac and sup.
Basecalling with sup still crashes the GPU and hac has no issue.

sup.log.gz
hac.log.gz

[malton-ont]

Thanks for the logs - can you please also upload logs for sup -b 32 and sup -b 0?

Cmake can be installed in a virtual environment using python3 - the current version there is 4.1.0, which should be more than sufficient.

python3 -m venv venv
. venv/bin/activate
pip install cmake

[awjga]

Hi @malton-ont ,

Thank you again for you help.
I have build and ran the CUDA test with default configuration successfully.

Here's the logs for the run with b0 and b32, both crashes the GPU

sup_b32.log.gz
sup_b0.log.gz

[malton-ont]

Hi @awjga,

Thanks for providing the logs. I find it interesting that the two show very different amounts for available memory:
sup_b0

[2025-09-11 07:48:49.845] [debug] cuda:0 memory available: 9.55GB

sup_b32

[2025-09-11 11:19:12.263] [debug] cuda:0 memory available: 5.78GB

This suggests there's something else taking up GPU memory on the device - if another process is allocating memory while dorado is running, this could cause issues? But I'd be surprised if this was occurring every single time, in which case there may be a bug in one of the CUDA kernels - are you able to share the data that is producing this crash?