Is there a solution&demo to use all features include PP+TP+DP+ZeRO?

[xinghua3632]

Any one knows?
Some demo use PP+TP+DP，but not use ZeRO optimizer, and some demo use ZeRO but not use PP

[binmakeswell]

Hi @xinghua3632
This is generally unnecessary. For example, ZeRO has divided the model, so it's not necessary to use PP+TP+DP at the same time.
You can test and select the most suitable solution based on your hardware and requirements.