Error invalid configuration argument at line 233 in file /src/csrc/ops.cu

[SzhangS]

11/22/2024 15:26:32 - INFO - main - ***** Running training *****
11/22/2024 15:26:32 - INFO - main - Num examples = 6
11/22/2024 15:26:32 - INFO - main - Num Epochs = 1667
11/22/2024 15:26:32 - INFO - main - Instantaneous batch size per device = 1
11/22/2024 15:26:32 - INFO - main - Total train batch size (w. parallel, distributed & accumulation) = 1
11/22/2024 15:26:32 - INFO - main - Gradient Accumulation steps = 1
11/22/2024 15:26:32 - INFO - main - Total optimization steps = 10000
11/22/2024 15:26:32 - INFO - main - #parameters of ReferenceNet is 859.520 M (840.710 M is trainable).
11/22/2024 15:26:32 - INFO - main - #parameters of Denoising U-Net is 1362.329 M (1362.329 M is trainable).
11/22/2024 15:26:32 - INFO - main - #parameters of V-Kps Guider is 1.086 M (1.086 M is trainable).
11/22/2024 15:26:32 - INFO - main - #parameters of Audio Projection is 29.524 M (29.524 M is trainable).
stage_3, Steps: 0%| | 0/10000 [00:00<?, ?it/s]Error invalid configuration argument at line 233 in file /src/csrc/ops.cu
训练时遇到了这个错误，请问该怎么解决？

[tiankuan93]

It seems like an issue with the training environment. I suggest you check the training environment.