GPU utilization is very low during training

[xin0623]

Thank you for your great work.
I am trying to replicate your results on Assembly101, but the GPU utilization is very low during training, which makes the training very slow. It takes more than an hour and a half to train one cycle on my RTX3060.
It seems to be a data I/O problem, but I haven't found a solution yet.
I would like to know what device you use for training and how long does one training cycle take?
I would be very grateful if you can reply

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01 Driver Version: 535.183.01 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 3060 Off | 00000000:01:00.0 On | N/A |
| 30% 45C P2 37W / 170W | 6224MiB / 12288MiB | 1% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

[ThibaultEllong]

Have you managed to fix it ? Training on dual GPU and it uses only 10% of those.