How to adjust learning rate and decay steps for parallel training in Pytorch backend?

[kechangming]

when using PyTorch for parallel training (e.g., on 4 GPUs), how should the learning rate, decay_step, and step be set so that the trend of the loss function's decrease is similar to that of single-GPU training?

[wanghan-iapcm]

the batch size in parallel training case indicates the batch size per each GPU.
therefore, the effective batch size at each training step is batch_size * number_of_gpus.

precisely reproduce the learning curve of single gpu is not trivial.