About automatic distributed parallelism in JAX #22054
Unanswered
south-ocean
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I see that Alpa can implement automatic distributed programs for Jax programs through decorator @parallel, which can implement Data Parallel, Tensor Parallel, and Pipeline Parallel. And the core idea of Alpa has been incorporated into XLA for a long time, so there is a decorator method like Alpa's Parallel in JAX to achieve automatic distributed parallelism? I've only seen pmap, vmap, etc., both of which are data parallel correlation.
Whether there is an automatic distributed data connection, or the JAX design concept is to let the developer choose the appropriate parallelization strategy according to the specific needs and scenarios, rather than the framework to provide automatic distributed parallelism. Thanks.
Beta Was this translation helpful? Give feedback.
All reactions