RuntimeError: CUDA error: an illegal memory access was encountered

[wrq9]

Thanks for your great work!
When I try to run python run.py demo_files/examples/fish.png --output-dir output/ --device cuda,
it shows:

/data/ricky/work/lib/python3.12/site-packages/albumentations/__init__.py:13: UserWarning: A new version of Albumentations is available: 2.0.5 (you have 1.4.15). Upgrade using: pip install -U albumentations. To disable automatic update checks, set the environment variable NO_ALBUMENTATIONS_UPDATE to 1.
  check_for_updates()
/data/ricky/work/lib/python3.12/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
  warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
Device used:  cuda:1
/data/ricky/work/lib/python3.12/site-packages/torch/functional.py:539: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:3637.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
Settings -> Mode=base, Device=cuda:1, Torchscript=disabled
  0%|                                                                                   | 0/1 [00:01<?, ?it/s]
Traceback (most recent call last):
  File "/data/ricky/stable-point-aware-3d/run.py", line 172, in <module>
    mesh, glob_dict = model.run_image(
                      ^^^^^^^^^^^^^^^^
  File "/data/ricky/stable-point-aware-3d/spar3d/system.py", line 605, in run_image
    meshes, global_dict = self.generate_mesh(
                          ^^^^^^^^^^^^^^^^^^^
  File "/data/ricky/stable-point-aware-3d/spar3d/system.py", line 776, in generate_mesh
    gb_pos = pos_bake[bake_mask]
             ~~~~~~~~^^^^^^^^^^^
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

My device:

RTX 4090 24G with CUDA 12.4

What should I do?