How many GPUS were used in the training process

[BBQLXDM]

Thank you for your work. I'd like to ask what kind of GPUs were used in the training process, how many of them were used, and how long the training process took.

[BBQLXDM]

你好我想问下训练过程使用了什么种类的GPU，多少个，训练时长有多长时间呢

[Hertin]

We use 8 gpus each with memory size of 80gb. The model needs about 20k step for audio loss to converge and needs another ~10k steps to achieve best performance.

[BBQLXDM]

We use 8 gpus each with memory size of 80gb. The model needs about 20k step for audio loss to converge and needs another ~10k steps to achieve best performance.

Thank you for your reply. May I ask how many 80G GPUs were used for stage1, stage2, and stage3 respectively, and how long did the training take for each stage?

[Hertin]

Stage 1 is trained by my colleague and I am not sure the exact number of GPUs used. What I know is that stage 1 took the longest time for about 5 to 6 days because of the large amount of speech data. Stage 2 and 3 each takes 1~2 days to train with 8 GPUs.

[BBQLXDM]

Stage 1 is trained by my colleague and I am not sure the exact number of GPUs used. What I know is that stage 1 took the longest time for about 5 to 6 days because of the large amount of speech data. Stage 2 and 3 each takes 1~2 days to train with 8 GPUs.

Thank you for your reply. At present, I only have 4 V100-32G GPUs. Is it possible to conduct the training for stage 2 and stage 3?

[Hertin]

I think it would be difficult. You will have to reduce batch size for the 7B model to fit in GPU memory.

[BBQLXDM]

I think it would be difficult. You will have to reduce batch size for the 7B model to fit in GPU memory.

thanks,i will try it