How to specify the process groups in 3D parallelism? #4063
Unanswered
Muggle666
asked this question in
Community | Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I am using 3D training parallelism, but how do I specify the process groups in different parallelism? Like I want to use tensor parallelism only within one node but use other parallelisms across nodes. Does colossal ai support this feature and how can I implement this? By the way, how does the colossal ai handle this by default?
Beta Was this translation helpful? Give feedback.
All reactions