Difference with torch.distributed

[base-y]

Hi, I am new to distributed training and am using huggingface to train large models. I see many options to run distributed training. Can I know what is the difference between the following options:

1. python train.py .....<ARGS>
2. python -m torch.distributed.launch <ARGS>
3. deepspeed train.py <ARGS>

I did not expect option 1 to use distributed training. But it even seem to use some sort of torch distributed training? In that case, whats the difference between option 1 and option 2?

Does deepspeed use torch.distributed in the background?

[base-y]

@tjruwase can you help me with this/

[tjruwase]

Historically, 1 was only capable of doing distributed training using a single multi-threaded process (1 thread per rank) and only worked within a node. I am not sure if that is still the case, or if it now defaults to 2 in the background.

2 was created for better distributed training using multiple processes (1 process per rank) and woks across nodes.

3 uses 2 in the background and adds some deepspeed specific options. Hope that helps.