Running hybrid mpi4py and Charm4py programs (mpi interop)

[matthiasdiener]

@juanjgalvez
Nov 30 2018

---

It is possible to run hybrid MPI/Charm4py programs. I have made an example (shown below) that mixes some MPI synchronous collectives with a non-trivial Charm workload. I have also been able to successfully run code that uses external MPI libraries, and have also run code that uses Isend/Irecv.

I have tested this on a Linux machine using OpenMPI. The requirement is to build Charm++ with the MPI layer. Here is an example of how to run the code:

$ git clone https://github.com/UIUC-PPL/charm4py
$ cd charm4py
$ git clone https://github.com/UIUC-PPL/charm charm_src/charm
$ python3 setup.py build_ext --inplace --mpi
$ export PYTHONPATH=`pwd`
$ mpirun -np 4 /usr/bin/python3 mpi-charm-hybrid.py

Here is the example code (updated for Charm4py v1.0):

# mpi-charm-hybrid.py
from mpi4py import MPI
from charm4py import charm, Chare, Group, Array, coro, Future, Channel, Reducer
import random

PARTICLES_PER_CELL_INIT = 300
CELLS_PER_PE = 4


class Cell(Chare):

    def __init__(self):
        # print('Cell', self.thisIndex, 'created')
        self.numParticles = PARTICLES_PER_CELL_INIT

	@coro
	def work(self, numCells, done_future):
        self.numCells = numCells
        self.done_future = done_future
        neighborA = Channel(self, remote=self.thisProxy[(self.thisIndex[0] - 1) % numCells])
        neighborB = Channel(self, remote=self.thisProxy[(self.thisIndex[0] + 1) % numCells])
        for _ in range(100):  # do 100 iterations
            # move random number of particles to my 2 neighbors
            a = random.randint(0, self.numParticles)
            self.numParticles -= a
            b = random.randint(0, self.numParticles)
            self.numParticles -= b
            neighborA.send(a)
            neighborB.send(b)
            self.numParticles += neighborA.recv()
            self.numParticles += neighborB.recv()
        # do a gather reduction of num particles, send result to element 0 of
        # my chare array
        self.reduce(self.thisProxy[0].calculateResults, self.numParticles,
                    Reducer.gather)


	def calculateResults(self, particles_in_cells):
        totalParticles = sum(particles_in_cells)
        minParticlesPerCell = min(particles_in_cells)
        avgParticlesPerCell = totalParticles / self.numCells
        maxParticlesPerCell = max(particles_in_cells)                  
        # send result to future
		self.done_future.send((minParticlesPerCell, avgParticlesPerCell,
                              maxParticlesPerCell, totalParticles))


class HybridMPICharmProgram(Chare):
	
	@coro
	def start(self):
        # this is the part that resembles an MPI program. but there is MPI and
        # charm code interspersed
        comm = MPI.COMM_WORLD
        myrank = comm.Get_rank()
        numRanks = comm.Get_size()
        print('Hello from MPI rank', myrank, 'numRanks=', numRanks)
        # test MPI barrier
        comm.Barrier()

        # test MPI broadcast
        test_bcast = None
        if myrank == 0:
            test_bcast = [4, 7, 12, 13]
        test_bcast = comm.bcast(test_bcast, root=0)
        assert test_bcast == [4, 7, 12, 13]

        # now test a charm workload
        numCells = charm.numPes() * CELLS_PER_PE
        if myrank == 0:
        	# create chare array of Cells
            cells = Array(Cell, numCells)
            # create future to wait for result of simulation
            f = Future()
            # start simulation
            cells.work(numCells, f)
            charm_result = f.get()
            # barrier among members of my group (see comment below)
            self.allreduce().get()
            # broadcast result of Charm simulation using MPI bcast
            comm.bcast(charm_result, root=0)
        else:
            # charm has to do the simulation using all PEs, so we can't have MPI
            # block the process. we do a self.allreduce() instead to wait for
            # rank 0. self.allreduce() returns a future and blocking on the future
            # only blocks this coroutine (other chares can continue doing work)
            self.allreduce().get()
            # get result of charm simulation using MPI broadcast
            charm_result = comm.bcast(None, root=0)
            assert int(charm_result[1]) == PARTICLES_PER_CELL_INIT
            assert charm_result[3] == PARTICLES_PER_CELL_INIT * numCells
            print('Rank', myrank, 'charm_result=', charm_result)

        comm.Barrier()
        if myrank == 0:
            exit()


def startMPI(args):
	Group(HybridMPICharmProgram).start()


charm.start(startMPI)

[matthiasdiener]

ccueto
Dec 03 2018

---

That’s looking great Juan, thanks for working on this so swiftly. I will try to test it on an example using charm4py and an MPI-based library by the end of this week, beginning of the next.

[matthiasdiener]

@juanjgalvez
Sep 15 2019

---

I have updated the example and instructions to run for Charm4py 1.0