Lifetime of pinned->unpinned objects in async native calls

[msedi]

CUDA has the ability of using streams which are essentially an asynchronous pipeline. Synchronous methods for example

CUresult cuMemcpy (CUdeviceptr dst, CUdeviceptr src, size_t ByteCount)

whereas the async method has

CUresult cuMemcpyAsync (CUdeviceptr dst, CUdeviceptr src, size_t ByteCount, CUstream hStream)

which gets an additional stream parameter. The stream (if used) parameter then flows from cuda function call to next where the stream can be seen as a waithandle where each function waits until the stream is signalled by the preceding method.

While this works well when working with unmanaged memory or value types it's getting a problem with unpinned managed memory.
Normally everytime you have to send memory to the GPU a pointer is involved that forces me to pin the data (mostly arrays of integral data types) and if the method is finished I can unpin it again. This works well for the synchronous case.

The async case is a bit more tricky since I don't know when the method as finished copying

float[] data = new float[]{1,2,3,4,5};
fixed(var *ptr = data)
{
  cuMemcpyAsync (dst, src, sizeof(float) * data.Length, hStream)
} // unpinned

I assume as the comment in the example above says that the array is unpinned at the end of the curly bracket. Which means the GC is free to move or event to collect the unused memory. While you can prevent the collection with GC.KeepAlive() the unpinned data is the thing that hurts more. Currently I haven't experienced problems because I'm not working with streams. But without using streams you are using performance.

There are also no callbacks where I get informed that I could unpin, bu even then I would have to work with GCHandle.Alloc() where I'm not sure in such a performance sensitive area.

Is there any solution or opinio of how to solve this?

[Clockwork-Muse]

These appear to both be native methods, so how are you calling them in C#?

Currently I haven't experienced problems because I'm not working with streams.

The method call is documented as being async anyways, so that you haven't had an issue yet is probably a minor miracle.

There are also no callbacks where I get informed that I could unpin,

From the look of the library, it appears that the intended pattern is to submit a bunch of work to a stream, then poll/wait on events or other related streams. If this were a game engine, you'd probably be checking at the start of each frame or something, and freeing/reusing based on that.

The async case is a bit more tricky

You can't used fixed for an async context. You'd have to do something similar to the sample for rule No. 10, which is how I'd recommend setting things up to create a wrapper library. That, or you're looking at allocating native memory and mapping it on the C# side as necessary (which might be the better option anyways).

[msedi]

@Clockwork-Muse. Thanks for your answer.

The method call is documented as being async anyways, so that you haven't had an issue yet is probably a minor miracle.

I was not quite detailed in my explanation, I'm currently not using the streams at all and therefore hadn't any problems. But more and more questions are coming up if stream can be used to improve performance.

You can't used fixed for an async context.

I could use GCHandle.Alloc but then the question is when to Free the memory. I agree qith your suggestions but my situation is a bit more complicated since our framework is based on either managed or unmanaged memory. The more I need to work with native interop and unmanaged memory and memory sizes beyond sizeof(int) >= 20TB it seems that C# isn't ready for that and the ways back to native pointers is the best way.

Span<T> has its problem that its a ref struct, Memory<T> has the problem that I need a MemoryManager<T>. In my case I would need an UnmanagedMemoryManager<T> which doesn't even exist in the framework (I have to create my own). But my data is stored in chunks of pages (not the ones from Windows).

I already had to go the way that I needed to call GC.AllocateArray() with pinned=true, because it is not possible to get a pointer from a Span if its not pinned (See also https://stackoverflow.com/questions/52190423/c-sharp-access-unmanaged-array-using-memoryt-or-arraysegmentt).

Nevertheless I think you are right I need to go with native memory in general which questions the whole use of C#/GC/managed memory in my case ;-)

[hez2010]

You may want to use NativeMemory.* APIs but you have to manage the memory on your own.

[msedi]

@hez2010. Yes and sometimes I don't have to because the data is already coming from memory mapped file views. But the problem is that are existing interfaces that allow managed and unmanaged memory depending on the task and the implementer. Additionally currently I do not need some IDiposable on the methods if I would work with unmanaged memory I would have to add it and per definition this is a breaking change. Lately I did a refactoring to look how many occurences would be influenced by the introduction of IDisposable and found that it would be around 3000 methods that would have to be reworked.

I need to check, as far as I have found stream now support callback, maybe I can insert some code there.