Stream Start and (not) receiving events

[ManickaP]

Based on the comment in remarks https://github.com/microsoft/msquic/blob/main/docs/api/StreamStart.md#remarks:

Important - No events are delivered on the stream until the app calls StreamStart (because of the race conditions that could occur) and it succeeds. This means that if the parent connection is shutdown (e.g. idle timeout or peer initiated) before calling StreamStart then the QUIC_STREAM_EVENT_SHUTDOWN_COMPLETE will not be delivered. So, apps that rely on that event to trigger clean up of the stream must handle the case where StreamStart is either not ever called or fails and clean up directly.

In what exact cases won't the events come? I've observed:

• If the StreamStart call fails
• If the START_COMPLETE event has error Status
• If the START_COMPLETE event is not PeerAccepted and PEER_ACCEPTED event hasn't arrived yet
  • With pending on the peer, I now see that the SHUTDOWN_COMPLETE comes after connection tear down. So the events do arrive, they are just held on not communicating with the peer in the meantime.

Are those all?

Just making sure we do proper clean up in all cases and don't wait for SHUTDOWN_COMPLETE if none will arrive.

[nibanks]

• If the StreamStart call fails, you will receive no notifications. It can only fail though if you're calling the API at the wrong time, with the wrong parameters, or on allocation failure.
• If the START_COMPLETE event indicates failure (non QUIC_SUCCEEDED(Status) code), then you will not get any further notification except for:
  • If you passed QUIC_STREAM_START_FLAG_SHUTDOWN_ON_FAIL, then you will get a SHUTDOWN_COMPLETE, and nothing else.
  • If you call StreamStart again (for instance, because you called the first time with QUIC_STREAM_START_FLAG_FAIL_BLOCKED and there wasn't any available), and it succeeds, then the stream will operate as normal.
• For all else, once the stream is successfully started, you may get other events any time afterwards, and are guaranteed to get the SHUTDOWN_COMPLETE as the last event on the stream.

Hopefully that answers all your questions, but I don't quite understand this statement:

So the events do arrive, they are just held on not communicating with the peer in the meantime.

As long as a stream hasn't been accepted by the peer, all the state is just "queued" locally. If the stream is eventually shutdown before the peer accepts it, everything is still cleaned up.

[ManickaP]

From my observation, if the stream received START_COMPLETE but is not PeerAccepted yet, you won't get SHUTDOWN_COMPLETE even if you ABORT_RECEIVE and shutdown write with GRACEFUL. The only thing that helped to unblock this was to call StreamShutdown(QUIC_STREAM_SHUTDOWN_FLAGS.ABORT | QUIC_STREAM_SHUTDOWN_FLAGS.IMMEDIATE), i.e. abort both sides in a single call.

[nibanks]

That's correct, and by design. The GRACEFUL flag indicates you are telling the peer all the data has been sent. You only get shutdown once that has been acknowledged. You can only get an ACK if you have sent it. You can only send it if your peer has accepted it. The same for abortive shutdown unless you use that IMMEDIATE flag.

[ManickaP]

To summarize this: let's assume we want to clean up the stream and we're using SHUTDOWN_ON_FAIL:

• If StreamStart fails, we should call StreamClose directly without waiting on any events. We can safely release context object after that, no events will ever arrive.
• If StreamStart succeeded and:
  • we aren't peer accepted yet (didn't get START_COMPLETE or PEER_ACCEPTED if pending on it), we need to abort both sides immediately with ABORT | IMMEDIATE and we'll get SHUTDOWN_COMPLETE immediately after which we can release the context object.
  • we are peer accepted, we can close gracefully and we'll get SHUTDOWN_COMPLETE eventually after which we can release the context object.

I'm trying to avoid the situation where we release the context object prematurely (may lead to free after use) or where we'll be waiting for SHUTDOWN_COMPLETE indefinitely (leak).

[nibanks]

For your "If StreamStart succeeded and" cases, I think it really depends. Just because the peer hasn't accepted the stream doesn't mean you should necessarily abort the stream. For instance, a local unidirectional stream, that an app just opens, starts, sends and gracefully shuts down is completely a valid scenario. For instance, this is an expected scenario for 0-RTT (before you actually start the connection).

Ideally, this should be an app-driven decision. If they want to abort, then abort. Otherwise it should wait for the stream to be accepted and all other events that follow.

[ManickaP]

In my particular case, we're not handing out the stream, this is a clean up from a "failed" stream open, which might happen due to user provided cancellation and not due to msquic. But once we hand out the stream, it's up to user to decide whether to abort or shutdown gracefully.