Training with pipeline parallelism example

[hpc-unex]

Hello!

I'm trying to train the example provided for pipeline parallelism in (https://github.com/hpcaitech/ColossalAI-Examples/tree/main/features/pipeline_parallel) for CIFAR10 and ResNet50.

I'm running a single node with 2 GPUs but something seems to be wrong with the execution since the accuracy is around 10%~ from the epoch 0 until the infinite. I'm using the "resnet.py" file from the repository with the only change that the processes are launched with mpi:

colossalai.launch(config=CONFIG,
host=None,
port=None,
backend='mpi',
rank = int(os.environ['OMPI_COMM_WORLD_RANK']),
world_size=int(os.environ['OMPI_COMM_WORLD_SIZE']),
local_rank = int(os.environ['OMPI_COMM_WORLD_LOCAL_RANK']),
seed=opt.manualSeed)

This configuration is tested in data and model parallelism and works correctly. Any idea? Someone have tested that example?

Thanks!

[YuliangLiu0306]

Hi, Unex
The pipeline parallelism feature is refactoring now, new pipeline parallelism will have following advantages:

1. Support more flexible dataflow: New PP module supports more than one tensor communication between stages and non-tensor data transporting.
2. Support more flexible graph partitioning: Instead of linked list style partitioning, you can split your module to DAG topology.
3. We will do more correctness tests and compatibility tests to make sure new module could work as expected.

[hpc-unex]

Thanks for your fast answer! The advantages sounds smooth. Do you know how long could it take?