Migrating training scripts to torchrun

[lkosh]

No description provided.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.74%. Comparing base (db6d0db) to head (06984f4).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1933      +/-   ##
==========================================
- Coverage   96.80%   96.74%   -0.06%     
==========================================
  Files         172      172              
  Lines        8442     8442              
==========================================
- Hits         8172     8167       -5     
- Misses        270      275       +5     

Flag	Coverage Δ
unittests	96.74% <ø> (-0.06%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[felixdittrich92]

Thanks 👍

Mh Normally in this case I think we can merge both scripts into one (DDP & the normal train script) ? - because the logic is the same - anyway what we should test is that the logging does still work with torchrun (W&B for example)

if args.backend:
   torch.cuda.set_device(rank)
   dist.init_process_group(backend=args.backend)