Skip to content

Commit be15836

Browse files
[float8 moe] Support TP (#1375)
We can remove this assertion, TP support for float8 rowwise MoE training was added in this PR stack: pytorch/ao#2473
1 parent 170b992 commit be15836

File tree

1 file changed

+0
-3
lines changed

1 file changed

+0
-3
lines changed

torchtitan/components/quantization/float8.py

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -59,9 +59,6 @@ def __init__(self, job_config: JobConfig, parallel_dims: ParallelDims):
5959

6060
# Validate MoE training prototype limitations.
6161
if self.moe_fqns:
62-
assert (
63-
job_config.parallelism.tensor_parallel_degree == 1
64-
), "Float8 MoE training prototype does not yet support tensor parallelism"
6562
assert (
6663
job_config.parallelism.pipeline_parallel_degree == 1
6764
), "Float8 MoE training prototype does not yet support pipeline parallelism"

0 commit comments

Comments
 (0)