New subsection 5: Local use of general datatypes, advice to users.

[softwaretraff]

Problem

It may be convenient to be able to copy MPI typed data locally on a process, for instance a matrix column (described as an
MPI vector) to a matrix row (MPI contiguous) or more sophisticated rearrangements of data described by MPI datatypes. There is no specific functionality in MPI for process local, typed memory copy. Possibly none is needed: Process local copy could be done over the MPI_COMM_SELF communicator. Add a small "advice to users" somewhere.

Proposal

Comment on the issue of process local, correctly MPI typed memory copy. Suggest a solution as "advice to users", possibly in new subsection in Chapter 5.

Changes to the Text

After Subsection 5.1.11 (possibly new 5.1.12): Process local use of general datatypes.

MPI derived datatypes can likewise describe the structure of data locally on the MPI processes. The need to copy possibly non-consecutive data from one process local data buffer to another process local data buffer, e.g., a column of a 2-dimensional matrix to a row of another 2-dimensional matrix, may arise in applications. Such a copy can be effected by having the process communicate with itself using the MPI_COMM_SELF communication. Collective or point-to-point communication (MPI_Sendrecv()) can be used.

Example: A column of an nxn matrix could be described by a vector datatype with n blocks, one element and stride n. Such a column can be copied into a contiguous vector using a contiguous rowtype of n elements using for instance a collective MPI_Allgather() operation:

MPI_Allgather(matrix,colltype,1,vector,rowtype,1,MPI_COMM_SELF);

Impact on Implementations

None, except the obligation that collectives or MPI_Sendrecv are fast with different datatypes in MPI_COMM_SELF.

Impact on Users

None. May help advanced users having had this problem.

References and Pull Requests

The issue is discussed with some benchmarking results in

Jesper Larsson Träff, Ioannis Vardas:
Library Development with MPI: Attributes, Request Objects, Group Communicator Creation, Local Reductions, and Datatypes. EuroMPI 2023: 5:1-5:10

In the paper, MPI_Sendrecv is recommended. It seems that for some/many MPI libraries, MPI_Allgather or MPI_Alltoall perform better.

[jeffhammond]

I think it would be really useful to have a MPI_Datatype (de)serialize function.

[devreal]

I think it would be really useful to have a MPI_Datatype (de)serialize function.

Something like MPI_PACK and MPI_UNPACK?

[jeffhammond]

i don't know if you're being cheeky but those functions to do not serialize and deserialize the datatype object itself, only a buffer described by one.

[softwaretraff]

Packed datatypes may contain meta information and you cannot do anything with them... You might need a signature type, see my EuroMPI 2021 paper:

Jesper Larsson Träff:
Signature Datatypes for Type Correct Collective Operations, Revisited. EuroMPI 2020: 81-88

A proposal could be to maintain a signature type implicitly together with derived datatypes. Some more work needed here.

Jesper

[devreal]

i don't know if you're being cheeky but those functions to do not serialize and deserialize the datatype object itself, only a buffer described by one.

Sorry, misunderstood what you are trying to achieve. I can see the value of serializing the datatype description but it adds constraints to future extensions (like custom callbacks for packing, which cannot be easily transferred to other processes).

[jeffhammond]

@devreal I don't know that I want the same thing as @softwaretraff - it just came up when I was thinking about this ticket.

[abouteiller]

I think it would be really useful to have a MPI_Datatype (de)serialize function.

I suppose you are referring to something different from TYPE_GET_ENVELOPE and TYPE_GET_CONTENTS you can use to inspect how the datatype is structured, but why would serializing the datatype itself be useful? Send it to another node?

[wgropp]

In both RMA and I/O, datatypes are applied to data at remote locations (MPI processes for RMA, file server for I/O). There are many ways to implement this; one way, particularly for more complex datatypes, is to ship a description of the MPI datatype to the target process and have the data moves executed there. Such MPI implementations will thus already have a way to serialize/deserialize an MPI datatype.