training stage 1

[Wh1t3zZwhite]

During training stage 1, it always gets stuck at a certain step. After debugging, I found that it is stuck at accelerator.backward(loss). At this point, the GPU usage is still very high. How should I resolve this?

[Wh1t3zZwhite]

Also, I found that this seems to be related to the seed I passed in, as it gets stuck at different steps depending on the seed.

[tiankuan93]

It seems like there might be an issue with the data. Could you please run the dataset separately to ensure that all the data is correct?

[Wh1t3zZwhite]

数据似乎可能存在问题。您能否单独运行数据集以确保所有数据都是正确的？

I tried using a single piece of data from the prepare_dataset folder as a test, but it still doesn't work. Could it be that there is an issue with the model I downloaded?

[tiankuan93]

Generally speaking, if you can successfully train for one step, it is likely not an issue with loading the model. You might want to check the loss values to see if there are any anomalies. Additionally, if the process consistently hangs at a specific step, it could be a multi-GPU communication issue. Have you tried waiting for about half an hour (depending on the timeout settings for multi-GPU communication) to see if any error messages appear?

[Wh1t3zZwhite]

It looks like the loss is correct:Loss value: 0.3705923557281494Loss value: 0.568477988243103
but after waiting for half an hour, the following error occurred:
[E ProcessGroupNCCL.cpp:475] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=2125, OpType=ALLREDUCE, NumelIn=198929280, NumelOut=198929280, Timeout(ms)=1800000) ran for 1800321 milliseconds before timing out.
[E ProcessGroupNCCL.cpp:489] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E ProcessGroupNCCL.cpp:495] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupNCCL.cpp:916] [Rank 1] NCCL watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=2125, OpType=ALLREDUCE, NumelIn=198929280, NumelOut=198929280, Timeout(ms)=1800000) ran for 1800321 milliseconds before timing out.
[2025-02-14 12:11:01,735] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 15844 closing signal SIGTERM
[2025-02-14 12:11:02,301] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -6) local_rank: 1 (pid: 15845) of binary: /root/anaconda3/envs/memo/bin/python

[tiankuan93]

First of all, this loss seems a bit high and it's unclear if it's normal. It needs attention (based on our previous experience, the average loss is around 0.1, with some fluctuations, and it can go up to around 0.3 at times). Additionally, the current error is most likely due to multi-GPU communication issues, but it could also be related to the training code. This requires detailed debugging.

[Wh1t3zZwhite]

First of all, this loss seems a bit high and it's unclear if it's normal. It needs attention (based on our previous experience, the average loss is around 0.1, with some fluctuations, and it can go up to around 0.3 at times). Additionally, the current error is most likely due to multi-GPU communication issues, but it could also be related to the training code. This requires detailed debugging.

In the model training setup, I have only changed the batch sizes, the number of GPUs used for training, and the model file path, but I am still encountering this issue. I am using two 3090 GPUs for training; is it because the GPUs are insufficient to support the training?

[tiankuan93]

First of all, this loss seems a bit high and it's unclear if it's normal. It needs attention (based on our previous experience, the average loss is around 0.1, with some fluctuations, and it can go up to around 0.3 at times). Additionally, the current error is most likely due to multi-GPU communication issues, but it could also be related to the training code. This requires detailed debugging.

In the model training setup, I have only changed the batch sizes, the number of GPUs used for training, and the model file path, but I am still encountering this issue. I am using two 3090 GPUs for training; is it because the GPUs are insufficient to support the training?

I can't be sure about this. You might need more detailed debugging to confirm the issue. Alternatively, you could try training with a single GPU first.

[Wh1t3zZwhite]

First of all, this loss seems a bit high and it's unclear if it's normal. It needs attention (based on our previous experience, the average loss is around 0.1, with some fluctuations, and it can go up to around 0.3 at times). Additionally, the current error is most likely due to multi-GPU communication issues, but it could also be related to the training code. This requires detailed debugging.

In the model training setup, I have only changed the batch sizes, the number of GPUs used for training, and the model file path, but I am still encountering this issue. I am using two 3090 GPUs for training; is it because the GPUs are insufficient to support the training?

I can't be sure about this. You might need more detailed debugging to confirm the issue. Alternatively, you could try training with a single GPU first.

I tried training with a single RTX 3090 GPU, but encountered VRAM insufficiency issues. It appears that one 3090 isn't sufficient for the training task. Nevertheless, I appreciate your assistance and wish you a pleasant day.

[2351548518]

What GPU was used for this training and how long did it take?