Skip to content

Commit 15bca3b

Browse files
rafbielsomarahmed1111
authored andcommitted
[CUDA] Fix race condition in CUDA stream creation
Do not increment NumComputeStreams / NumTransferStreams before cuStreamCreateWithPriority returns. Too early increment caused other threads to read the incremented count before a CUDA stream was created and try to use an invalid stream handle, causing crashes. The construction: ``` if (condition) { lock_this_scope if (condition) { create_object update_condition } } use_object ``` is only thread-safe if update_condition happens after create_object is completed.
1 parent a96fcbc commit 15bca3b

File tree

1 file changed

+4
-2
lines changed

1 file changed

+4
-2
lines changed

source/adapters/cuda/queue.cpp

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,8 @@ CUstream ur_queue_handle_t_::getNextComputeStream(uint32_t *StreamToken) {
4545
// change NumComputeStreams after that
4646
if (NumComputeStreams < ComputeStreams.size()) {
4747
UR_CHECK_ERROR(cuStreamCreateWithPriority(
48-
&ComputeStreams[NumComputeStreams++], Flags, Priority));
48+
&ComputeStreams[NumComputeStreams], Flags, Priority));
49+
++NumComputeStreams;
4950
}
5051
}
5152
Token = ComputeStreamIndex++;
@@ -110,7 +111,8 @@ CUstream ur_queue_handle_t_::getNextTransferStream() {
110111
// change NumTransferStreams after that
111112
if (NumTransferStreams < TransferStreams.size()) {
112113
UR_CHECK_ERROR(cuStreamCreateWithPriority(
113-
&TransferStreams[NumTransferStreams++], Flags, Priority));
114+
&TransferStreams[NumTransferStreams], Flags, Priority));
115+
++NumTransferStreams;
114116
}
115117
}
116118
uint32_t StreamI = TransferStreamIndex++ % TransferStreams.size();

0 commit comments

Comments
 (0)