Check on RMA restrictions on using locks

[schulzm]

Problem

Check on RMA restrictions on using locks - this came up during the December 2020 meeting

Proposal

tbd

Changes to the Text

tbd

Impact on Implementations

tbd

Impact on Users

tbd

References

Insert any internal (other issues) or external (websites, papers, etc.) references here.

[devreal]

@schulzm Do you have details on this? I must have missed that discussion...

[wgropp]

We now allow locks on all windows except those created with MPI_WIN_CREATE. And since we allow locks on memory attached with MPI_WIN_ATTACH, we allow locks on all memory - an implementation is expected to handle passive target on any kind of memory. The limitation of MPI_WIN_ATTACH is a relic of the MPI-2 design.

[jeffhammond]

We now allow locks on all windows except those created with MPI_WIN_CREATE.

I cannot come up with any explanation of how this statement could be correct.

MPI_Win_lock and MPI_Win_unlock have been permitted and used on windows constructed via MPI_Win_create since MPI-2 days, and I have used them billions of times with a wide range of implementations, in the context of ARMCI-MPI and NWChem.

To prove my point, https://www.mpi-forum.org/docs/mpi-2.2/mpi22-report/node231.htm contains the following:

no_locks --- if set to true, then the implementation may assume that the local window is never locked (by a call to MPI_WIN_LOCK). This implies that this window is not used for 3-party communication, and RMA can be implemented with no (less) asynchronous agent activity at this process.

How can there be an info key to tell the implementation something will not be used, if that thing is not permitted to be used?

[dholmes-epcc-ed-ac-uk]

I think both statements/assertions can be justified - @wgropp is correct de jure and @jeffhammond is correct de facto.

We can chart the progression of the de jure permitted restriction to which @wgropp refers from MPI-2.2, through MPI-3.1, to MPI-4.0:

MPI-2.2, §11.4.3, p358, lines 13-15: "Implementors may restrict the use of RMA communication that is synchronized by lock calls to windows in memory allocated by MPI_ALLOC_MEM (Section 8.2, page 274). Locks can be used portably only in such memory."

MPI-3.1, §11.5.3, p 448, lines 1-4: "Implementors may restrict the use of RMA communication that is synchronized by lock calls to windows in memory allocated by MPI_ALLOC_MEM (Section 8.2), MPI_WIN_ALLOCATE (Section 11.2.2), or attached with MPI_WIN_ATTACH (Section 11.2.4). Locks can be used portably only in such memory."

MPI-4.0-RC-Nov20, §12.5.3, p595, lines 9-13: "Implementors may restrict the use of RMA communication that is synchronized by lock calls to windows in memory allocated by MPI_ALLOC_MEM (Section 9.2), MPI_WIN_ALLOCATE (Section 12.2.2), MPI_WIN_ALLOCATE_SHARED (Section 12.2.3), or attached with MPI_WIN_ATTACH (Section 11.2.4). Locks can be used portably only in such memory."

In practice, I have heard the de facto position espoused by @jeffhammond described as "if an MPI implementation must support RMA using any bit of memory that the user decides to attach to a dynamic window, then surely it can support RMA in a window created with MPI_WIN_CREATE" and "the rationale following this excerpt no longer makes sense because it assumes that window memory is 'shared' and that no software agent is needed precisely because it is shared, however there is no guarantee that memory attached to a dynamic window is shared."

It is this kind of disparity between de jure and de facto interpretation of the MPI Standard that the semantic terms WG attempted to address in the recent update to the Terms chapter. We did not touch the RMA chapter; that is a TODO item, although probably for the RMA WG rather than the Semantic Terms WG.

[wgropp]

Dan is correct, and captures what I meant when I wrote this. A more precise statement is that there are no restrictions on the memory that may be used with MPI_WIN_LOCK/UNLOCK except for windows created with MPI_WIN_CREATE, where the MPI implementation may require the use of MPI_ALLOC_MEM for the window memory in that case. That is what the standard currently says, and my point was exactly that this is no longer a useful restriction for implementations. But is is still what the standard says, and we're not talking about the extensions commonly implemented by implementations, we are discussing the standard.

[dholmes-epcc-ed-ac-uk]

I would also follow up with the reasonable/legitimate question: "if there is no restriction (except when there is), what happens when my program does the bad thing (that isn't)?"

Can I rely on it silently working, just slowly? Is it an erroneous program that gives unspecified or even undefined behaviour? Am I guaranteed to get an MPI error/exception, or an abort, or is it always fatal?

[jeffhammond]

The right thing to do for MPI users here is require that when an implementation requires MPI_ALLOC_MEM when MPI_WIN_CREATE is used with MPI_WIN_LOCK, that the implementation return an error when the user calls MPI_WIN_CREATE without the no_locks info key. That is when the implementation can detect the problem and ensures that MPI applications will not continue to execute in a state where things will not work as expected.

I do not see an error code that makes sense here, so I would suggest that something like MPI_ERR_RMA_MEM or MPI_ERR_RMA_ALLOC be used.

[dholmes-epcc-ed-ac-uk]

@jeffhammond that proposal is not backwards compatible (ducks to avoid heavy thrown object).

If the decision is that the restriction is needed and must be kept (a big "if" there, I suspect that the restriction will instead be dropped), then may I suggest instead that an info key/value is set by MPI during the MPI_WIN_CREATE call?
Specifically, MPI could be required to set the no_locks info key during the call if and only if it does not support locks for that type of window. An alert user could then write new code to check that info key and avoid actually calling MPI_WIN_LOCK by discovering that it is not permitted because MPI has asserted no_locks on their behalf.

If the user supplied MPI_INFO_NULL to the MPI_WIN_CREATE call, then they would have to subsequently use MPI_WIN_GET_INFO to discover and examine this info key. If the user tried to change the value of no_locks using MPI_WIN_SET_INFO then the value would be left unchanged by the call (not an error, just a waste of cycles).

Any MPI-3.1 program would still execute fine (unchanged), even if it relied on an implementation-specific extension to the MPI Standard that supported locks for windows created via MPI_WIN_CREATE.
The programmer of such a non-portable application could gain additional portability (from e.g. MPI-4.1 onwards) by adding new code to check the set-by-MPI info key and handling the absence of the extension when that occurs.

[jeffhammond]

I don't care about the backwards incompatibility issue because we are only talking about implementations that had severe restrictions on the behavior of RMA.

Removing the restriction is also fine, because the methods for implementation RMA have evolved a great deal over the past 20 years and I'm not aware of any good reasons to have it.

[devreal]

A quick recap of the RMA WG discussion on 9/14/2021 on this:

• There may be memory that MPI implementations cannot register with the NIC on today's systems (e.g., GPU unified memory) in a call to MPI_Win_create.
• With active target, the implementation has the guarantee that all targets and origins will end up in collective synchronization calls and operations can be locally accumulated until then, making the implementation easy.
• With passive target, the implementation has to fall back to some emulation (e.g., active messages) to perform the operations
  • There was some uncertainty whether the target of a put is allowed to poll on the local memory using loads, waiting for the operation to complete without calling into MPI. Looking at the language in §12.7.3 I believe such an application would be erroneous (i.e., applications are required to call into MPI at some point to progress outstanding RMA communication):

  Suppose that in the last example the send-receive pair is replaced by a write-to-socket/read-from-socket pair. Then MPI does not specify whether deadlock is avoided.

  • As a broader issue, however, we should discuss whether such emulation is desirable since the performance likely does not meet the user's expectations.
• We discussed whether we could move the failure point from MPI_Win_lock to MPI_Win_create if the no_lock info key is not present and memory that cannot be registered was passed. This would give users a way to treat such cases in a collective situation (versus a point in time where application failure is likely the only remedy). However, this would likely break most existing applications, which do not expect either an error being raised or MPI_WIN_NULL being returned.

Take-away: if we remove the limitation on memory that can be used with locks we force implementations to guarantee a fall-back to an emulation scheme if the memory is not suitable. My guess is that all implementations have such capabilities but we should discuss whether this behavior is desirable or whether the potentially sub-par performance ends up hurting applications.