Generalized RMA Atomics

[pavanbalaji]

Problem

The MPI-3.1 standard is unclear on which accumulate-style operations are atomic with respect to each other. In particular, by default, we say that "same op" and "no op" are atomic with respect to each other. But this does not specify:

1. If two different functions using the same MPI_Op are atomic with respect to each other.

2. If compare-and-swap (which does not have an MPI_Op argument) is atomic with respect to any of the other accumulate-style operations.

An additional problem is that saying that "same op" is atomic does not tell the MPI implementation what that operation is. MPI implementations can benefit from knowing what that operation is in order to decide what operations can be done, for example, in hardware, while maintaining atomicity.

Proposal

This proposal makes two major changes:

1. It changes the default behavior of MPI to be atomic with respect to all accumulate-style operations instead of just same_op_no_op. This sets the default to the most conservative behavior and allows users to weaken such guarantees by using info keys, which is the same model that we use everywhere else in the standard.

2. It allows users to specify exactly which operations they would use instead of simply saying same_op_no_op, which is more helpful for implementers to optimize performance.

Impact on Implementations

In MPI-3.1, implementations were only required to maintain atomicity for same_op_no_op. This proposal changes the default to be atomic for all accumulate-style operations.

The additional info keys gives more room for performance optimization, which can be used by implementations, if desired.

Impact on Users

Some cases that could have been specified with the old key can no longer be specified with the new key. However, we expect the impact on applications to be minimal to non-existent.

Final Proposal PDF

one-side-2.pdf

Pull Request

https://github.com/mpi-forum/mpi-standard/pull/93

[pavanbalaji]

Cc: @minsii, @jeffhammond, @hjelmn, @rsth.

[minsii]

The which_accumulate_ops paragraph looks good to me except sentence:Specifying a value allows all MPI routines using that operation to be included in the group of atomic operations. MPI collective routines may also use the operations, but they should not be included in atomics.

Besides, in the first page of the one-sided chapter, it defines:
This chapter refers to an operations set that includes all remote update, remote read and update, and remote atomic swap operations as “accumulate” operations.

However, we use "accumulates operations" as the MPI_op operation in the which_accumulate_ops paragraph.

[pavanbalaji]

I wish you had joined the call today because we had planned to discuss this proposal there. Can you send your proposed updated text? I'll send out another note to the Forum. This is due today.

[minsii]

Sorry, somehow I overlooked the meeting and github message...

I guess it is too late for updated text. Here are my proposed changes:

1. Use "operations used in the accumulate routines" instead of confusing "accumulate operation"
2. Use "MPI accumulate routines" instead of "MPI routines"

which_accumulate_ops — specifies which operations used in the accumulate routines are expected to be basic-element-wise atomic with respect to each other, as a comma-separated list. Valid values for this key are based on the predefined operations for MPI_REDUCE, as well as MPI_REPLACE and MPI_NO_OP. For example, MPI_SUM can be specified with the value string sum. Specifying a value allows all MPI accumulate routines using that operation to be included in the group of atomic operations. MPI_COMPARE_AND_SWAP is an exception to this rule because it does not take an MPI_Op argument; one can include this routine in the list of atomic operations by using the value compare_and_swap. A special info value, all, can be used to specify all predefined operations defined for MPI_REDUCE, replace, no_op, and compare_and_swap. The default is all.

[minsii]

Valid info values are decapitalized and deprefixed versions of the predefined operations listed in Section 5.9.2 (e.g., sum for specifying all accumulate routines with the MPI_SUM operator), replace (for all accumulate routines with the MPI_REPLACE operator), no_op (for all accumulate routines with the MPI_NO_OP operator), and compare_and_swap (for the MPI_COMPARE_AND_SWAP routine).
seems problematic to me. MPI_REPLACE, MPI_NO_OP and MPI_COMPARE_AND_SWAP are not defined in Section 5.9.2.

How about:
Valid info values are decapitalized and deprefixed versions of the predefined operations listed in Section 5.9.2 for MPI_REDUCE, as well as MPI_REPLACE, MPI_NO_OP, and MPI_COMPARE_AND_SWAP. For example, MPI_SUM can be specified with the value string sum, MPI_NO_OP can be specified with the value string no_op. MPI_COMPARE_AND_SWAP is a special value because it does not take an MPI_Op argument; one can include this routine in the list of atomic operations by using the value compare_and_swap.

[minsii]

Please ignore the above comment. I misunderstood the new text. It looks OK to me.

[pavanbalaji]

The reason that I removed MPI_REDUCE was because other operations can use them too.

If MPI_REPLACE, etc., were already defined in that table, I wouldn't have added them in the list above. I was trying to say "everything in that table" + these things. I realize the ambiguity in the English language, but I don't think it creates any ambiguity in the standard.

[pavanbalaji]

I've listed a single version of the proposal pdf on the top and deleted all other versions to avoid confusion.

[jsquyres]

@pavanbalaji Where is the PR for this PDF? I see green and red text on page 4 -- which is old, which is new? I'd like to see the PR where I can read the latex diff. Thanks.

[pavanbalaji]

@pavanbalaji Where is the PR for this PDF? I see green and red text on page 4 -- which is old, which is new? I'd like to see the PR where I can read the latex diff. Thanks.

Updated the above description to point to the PR. Red is delete. Green is add.