About sample example to send multiple data

[murphypei]

I have read all the stream docs and the sample demo, still have some confusion.

In the sample demo, what I think is that the client calls StreamStart, and the server gets QUIC_CONNECTION_EVENT_PEER_STREAM_STARTED, then the client send the SendBuffer, and the server gets QUIC_STREAM_EVENT_RECEIVE, but there is only one send.

I think that if I want to send data continuously, I just need create and send buffers continuously like these and the server will gets QUIC_STREAM_EVENT_RECEIVE and processes data continuously.

the ClientSend function

  // Starts the bidirectional stream. By default, the peer is not notified of
  // the stream being started until data is sent on the stream.
  //
  if (QUIC_FAILED(
          Status = MsQuic->StreamStart(Stream, QUIC_STREAM_START_FLAG_IMMEDIATE))) {
    printf("StreamStart failed, 0x%x!\n", Status);
    MsQuic->StreamClose(Stream);
    goto Error;
  }

  //
  // Allocates and builds the buffer to send over the stream.
  //
  while (1) {
    cv::Mat frame;
    if (!cap.read(frame)) // if not success, break loop
    {
      break;
    }

    auto totalSize = frame.total() * frame.elemSize();
    auto SendBufferRaw = (uint8_t *)malloc(totalSize + sizeof(QUIC_BUFFER));
    if (SendBufferRaw == NULL) {
      printf("SendBuffer allocation failed!\n");
      Status = QUIC_STATUS_OUT_OF_MEMORY;
      goto Error;
    }
    auto SendBuffer = (QUIC_BUFFER *)SendBufferRaw;
    SendBuffer->Buffer = SendBufferRaw + sizeof(QUIC_BUFFER);
    SendBuffer->Length = totalSize;
    memcpy(SendBuffer->Buffer, frame.data, totalSize);

    printf("[strm][%p] Sending frame:%d, length:%lu, addr:%p...\n", Stream,
           ++frameCount, totalSize, SendBuffer->Buffer);

    //
    // Sends the buffer over the stream. Note the FIN flag is passed along with
    // the buffer. This indicates this is the last buffer on the stream and the
    // the stream is shut down (in the send direction) immediately after.
    //
    if (QUIC_FAILED(Status = MsQuic->StreamSend(Stream, SendBuffer, 1,
                                                QUIC_SEND_FLAG_ALLOW_0_RTT,
                                                SendBuffer))) {
      printf("StreamSend failed, 0x%x!\n", Status);
      free(SendBufferRaw);
      goto Error;
    }
  }

the ServerStreamCallback function

case QUIC_STREAM_EVENT_RECEIVE:
    //
    // Data was received from the peer on the stream.
    //
    printf("[strm][%p] Server stream data received: buffer_count:%d, "
           "total_buffer_length:%lu, ServerRecvCount:%lu\n",
           Stream, Event->RECEIVE.BufferCount, Event->RECEIVE.TotalBufferLength,
           ServerRecvCount);
    for (uint32_t i = 0; i < Event->RECEIVE.BufferCount; ++i) {
      memcpy(ServerRecvBuffer + ServerRecvCount,
             Event->RECEIVE.Buffers[i].Buffer,
             Event->RECEIVE.Buffers[i].Length);
      ServerRecvCount += Event->RECEIVE.Buffers[i].Length;
    }

    if (ServerRecvCount >= CacheSizePerFrame) {
      cv::Mat img = cv::Mat(CapHeight, CapWidth, CV_8UC3, ServerRecvBuffer);
      cv::imshow(ShowWindowName, img);
      if (cv::waitKey(10) == 'q') {
        break;
      }
      memmove(ServerRecvBuffer, ServerRecvBuffer + CacheSizePerFrame,
              ServerRecvCount - CacheSizePerFrame);
      ServerRecvCount -= CacheSizePerFrame;
    }

    return QUIC_STATUS_CONTINUE;

But the code didn't work, the server start receive data after all data is sent. As docs says:

There are two buffering modes for sending supported by MsQuic. The first mode has MsQuic buffer the stream data internally. As long as there is room to buffer the data, MsQuic will copy the data locally and then immediately complete the send back to the app, via the QUIC_STREAM_EVENT_SEND_COMPLETE event. If there is no room to copy the data, then MsQuic will hold onto the buffer until there is room.

I know the client calls StreamSend is immediately complete, but even if I have enough frames to send, the server still couldn't receive any partial data. So did I make any mistakes? Any suggestions will help me, thanks.

[murphypei]

I have found that the client start print the sending log, but the server didn't get connected. After the client sent all frames with printing the log [strm][0x7fa91824eda0] Sending frame ***, then the server print the log [conn][0x7f4010000b60] Server connected and start receive data.

[murphypei]

@nibanks Could you please provide any suggestion about my question?

[nibanks]

Please verify you are getting the QUIC_CONNECTION_EVENT_PEER_STREAM_STARTED event on the server. I am willing to guess that you didn't configure PeerBidiStreamCount or PeerUnidiStreamCount settings on the server, to allow for the streams to be created.

[nicely-dev]

I have the same problem (and using datagrams shows the same behavior). The additional messages are only sent when closing the connection! The StreamSend documentation mentions "The app may call StreamSend (on any stream) with a null/empty buffer with QUIC_SEND_FLAG_DELAY_SEND unset to force a flush." but this does not force the flush in my attempts. Please advise.

[nibanks]

You will need to grab the logs to debug what is going on.

[nicely-dev]

Hi Nick, there should be a single stream in the capture. A first message is sent and a reply is received @18:05:56 (I always double the calls with a MsQuic->StreamSend with a NULL buffer as my reading let me to believe that it could flush the buffers but it does not seem to change anything and I can redo without this if it helps). After a few seconds I try to send 3 messages (@18:06:06) which are not received on the server side (and a wireshark shows no activity). When closing the app @18:06:16 I then get a lot of activity...
Extract of this period here:

[18]1F34.0B50::2024/03/27-18:05:56.239764400 [Microsoft-Quic][pack][16492674416642] Encrypting
[18]1F34.0B50::2024/03/27-18:05:56.239768800 [Microsoft-Quic][pack][16492674416642] Finalizing
[18]1F34.0B50::2024/03/27-18:05:56.239769100 [Microsoft-Quic][conn][0x24CA5E7BA50][TX][7] 5(ONE_RTT) (1392 bytes)
[18]1F34.0B50::2024/03/27-18:05:56.239770600 [Microsoft-Quic][conn][0x24CA5E7BA50] Sending batch. 2 datagrams
[18]1F34.0B50::2024/03/27-18:05:56.239770900 [Microsoft-Quic][data][0x24CA5E7D9A0] Send 1392 bytes in 1 buffers (segment=0) Dst=[::1]:56789 Src=[::1]:49517
[18]1F34.0B50::2024/03/27-18:05:56.239778200 [Microsoft-Quic][pack][16492674416642] Batch sent
[18]1F34.0B50::2024/03/27-18:05:56.239778900 [Microsoft-Quic][conn][0x24CA5E7BA50] Flush complete flags=0x0
[18]1F34.0B50::2024/03/27-18:05:56.239779200 [Microsoft-Quic][conn][0x24CA5E7BA50] Execute: 3(FLUSH_STREAM_RECV)
[18]1F34.0B50::2024/03/27-18:05:56.239779900 [Metadata]Metadata: TraceEventInfo, Provider=Microsoft-Quic
[18]1F34.0B50::2024/03/27-18:05:56.239779900 [Microsoft-Quic][strm][0x24CA5EC7010] Indicating QUIC_STREAM_EVENT_RECEIVE [48 bytes, 1 buffers, 0 flags]
[14]1F34.0B50::2024/03/27-18:06:06.071830800 [Microsoft-Quic][ api] Enter 22(STREAM_SEND) (0x24CA5EC7010).
[14]1F34.0B50::2024/03/27-18:06:06.071832400 [Microsoft-Quic][strm][0x24CA5EC7010] App queuing send [560 bytes, 1 buffers, 3 flags]
[14]1F34.0B50::2024/03/27-18:06:06.071833300 [Microsoft-Quic][strm][0x24CA5EC7010] Send Blocked Flags: 0
[14]1F34.0B50::2024/03/27-18:06:06.071836400 [Microsoft-Quic][strm][0x24CA5EC7010] Send Request [0000024CA5EC93E0] queued with 560 bytes at offset 152 (flags 0x3)
[14]1F34.0B50::2024/03/27-18:06:06.071837000 [Microsoft-Quic][strm][0x24CA5EC7010] Setting flags 0x10 (existing flags: 0x0)
[14]1F34.0B50::2024/03/27-18:06:06.071837800 [Microsoft-Quic][conn][0x24CA5E7BA50] Queueing send flush, reason=1(STREAM_FLAGS)
[14]1F34.0B50::2024/03/27-18:06:06.071839200 [Microsoft-Quic][strm][0x24CA5EC7010] SF:10 FC:524288 QS:712 MAX:152 UNA:152 NXT:152 RECOV:0-0 REL: 0
[14]1F34.0B50::2024/03/27-18:06:06.071839600 [Microsoft-Quic][ api] Exit 459749
[14]1F34.0B50::2024/03/27-18:06:06.071841500 [Microsoft-Quic][ api] Enter 22(STREAM_SEND) (0x24CA5EC7010).
[14]1F34.0B50::2024/03/27-18:06:06.071841900 [Microsoft-Quic][strm][0x24CA5EC7010] App queuing send [0 bytes, 1 buffers, 3 flags]
[14]1F34.0B50::2024/03/27-18:06:06.071842800 [Microsoft-Quic][strm][0x24CA5EC7010] Send Request [0000024CA5EC8370] queued with 0 bytes at offset 712 (flags 0x3)
[14]1F34.0B50::2024/03/27-18:06:06.071843700 [Microsoft-Quic][strm][0x24CA5EC7010] SF:10 FC:524288 QS:712 MAX:152 UNA:152 NXT:152 RECOV:0-0 REL: 0
[14]1F34.0B50::2024/03/27-18:06:06.071844000 [Microsoft-Quic][ api] Exit 459749
[10]1F34.97DC::2024/03/27-18:06:06.162747500 [Microsoft-Quic][ api] Enter 18(STREAM_OPEN) (0x24CA5E7BA50).
[10]1F34.97DC::2024/03/27-18:06:06.162749900 [Microsoft-Quic][strm][0x24CA5EC9C80] Allocated, Conn=0x24CA5E7BA50
[10]1F34.97DC::2024/03/27-18:06:06.162755100 [Microsoft-Quic][ api] Exit 0
[10]1F34.97DC::2024/03/27-18:06:06.162755700 [Microsoft-Quic][ api] Enter 20(STREAM_START) (0x24CA5EC9C80).
[10]1F34.97DC::2024/03/27-18:06:06.162757600 [Microsoft-Quic][ api] Exit 459749
[10]1F34.97DC::2024/03/27-18:06:06.162758100 [Microsoft-Quic][ api] Enter 22(STREAM_SEND) (0x24CA5EC9C80).
[10]1F34.97DC::2024/03/27-18:06:06.162758800 [Microsoft-Quic][strm][0x24CA5EC9C80] App queuing send [272 bytes, 1 buffers, 3 flags]
[10]1F34.97DC::2024/03/27-18:06:06.162759700 [Microsoft-Quic][ api] Exit 459749
[6]1F34.0B50::2024/03/27-18:06:07.095681100 [Microsoft-Quic][ api] Enter 22(STREAM_SEND) (0x24CA5EC7010).
[6]1F34.0B50::2024/03/27-18:06:07.095684600 [Microsoft-Quic][strm][0x24CA5EC7010] App queuing send [560 bytes, 1 buffers, 3 flags]
[6]1F34.0B50::2024/03/27-18:06:07.095688700 [Microsoft-Quic][strm][0x24CA5EC7010] Send Request [0000024CA5ECB0F0] queued with 560 bytes at offset 712 (flags 0x3)
[6]1F34.0B50::2024/03/27-18:06:07.095690000 [Microsoft-Quic][strm][0x24CA5EC7010] SF:10 FC:524288 QS:1272 MAX:152 UNA:152 NXT:152 RECOV:0-0 REL: 0
[6]1F34.0B50::2024/03/27-18:06:07.095690900 [Microsoft-Quic][ api] Exit 459749
[6]1F34.0B50::2024/03/27-18:06:07.095693000 [Microsoft-Quic][ api] Enter 22(STREAM_SEND) (0x24CA5EC7010).
[6]1F34.0B50::2024/03/27-18:06:07.095693500 [Microsoft-Quic][strm][0x24CA5EC7010] App queuing send [0 bytes, 1 buffers, 3 flags]
[6]1F34.0B50::2024/03/27-18:06:07.095694400 [Microsoft-Quic][strm][0x24CA5EC7010] Send Request [0000024CA5ECB140] queued with 0 bytes at offset 1272 (flags 0x3)
[6]1F34.0B50::2024/03/27-18:06:07.095695300 [Microsoft-Quic][strm][0x24CA5EC7010] SF:10 FC:524288 QS:1272 MAX:152 UNA:152 NXT:152 RECOV:0-0 REL: 0
[6]1F34.0B50::2024/03/27-18:06:07.095695600 [Microsoft-Quic][ api] Exit 459749
[12]1F34.0B50::2024/03/27-18:06:08.019227400 [Microsoft-Quic][ api] Enter 22(STREAM_SEND) (0x24CA5EC7010).
[12]1F34.0B50::2024/03/27-18:06:08.019230700 [Microsoft-Quic][strm][0x24CA5EC7010] App queuing send [560 bytes, 1 buffers, 3 flags]
[12]1F34.0B50::2024/03/27-18:06:08.019235400 [Microsoft-Quic][strm][0x24CA5EC7010] Send Request [0000024CA5ECB190] queued with 560 bytes at offset 1272 (flags 0x3)
[12]1F34.0B50::2024/03/27-18:06:08.019236700 [Microsoft-Quic][strm][0x24CA5EC7010] SF:10 FC:524288 QS:1832 MAX:152 UNA:152 NXT:152 RECOV:0-0 REL: 0
[12]1F34.0B50::2024/03/27-18:06:08.019237400 [Microsoft-Quic][ api] Exit 459749
[12]1F34.0B50::2024/03/27-18:06:08.019240400 [Microsoft-Quic][ api] Enter 22(STREAM_SEND) (0x24CA5EC7010).
[12]1F34.0B50::2024/03/27-18:06:08.019240900 [Microsoft-Quic][strm][0x24CA5EC7010] App queuing send [0 bytes, 1 buffers, 3 flags]
[12]1F34.0B50::2024/03/27-18:06:08.019241800 [Microsoft-Quic][strm][0x24CA5EC7010] Send Request [0000024CA5ECB1E0] queued with 0 bytes at offset 1832 (flags 0x3)
[12]1F34.0B50::2024/03/27-18:06:08.019242700 [Microsoft-Quic][strm][0x24CA5EC7010] SF:10 FC:524288 QS:1832 MAX:152 UNA:152 NXT:152 RECOV:0-0 REL: 0
[12]1F34.0B50::2024/03/27-18:06:08.019243100 [Microsoft-Quic][ api] Exit 459749
[12]1F34.0B50::2024/03/27-18:06:16.060552600 [Metadata]Metadata: TraceEventInfo, Provider=Microsoft-Quic
[12]1F34.0B50::2024/03/27-18:06:16.060552600 [Microsoft-Quic][strm][0x24CA5EC7010] Receive complete [48 bytes]
[12]1F34.0B50::2024/03/27-18:06:16.060554600 [Microsoft-Quic][conn][0x24CA5E7BA50] Execute: 4(FLUSH_SEND)
[12]1F34.0B50::2024/03/27-18:06:16.060556600 [Microsoft-Quic][conn][0x24CA5E7BA50] Flushing Send. Allowance=12000 bytes
[12]1F34.0B50::2024/03/27-18:06:16.060558100 [Microsoft-Quic][pack][16492674416642] Created in batch 16492674416642
[12]1F34.0B50::2024/03/27-18:06:16.060559200 [Microsoft-Quic][strm][0x24CA5EC7010] Writing frames to packet 16492674416642
[12]1F34.0B50::2024/03/27-18:06:16.060562300 [Microsoft-Quic][strm][0x24CA5EC7010] Built stream frame, offset=152 len=1281 fin=0
[12]1F34.0B50::2024/03/27-18:06:16.060563700 [Microsoft-Quic][strm][0x24CA5EC7010] SF:10 FC:524288 QS:1832 MAX:1433 UNA:152 NXT:1433 RECOV:0-0 REL: 0
[12]1F34.0B50::2024/03/27-18:06:16.060564500 [Microsoft-Quic][strm][0x24CA5EC7010]   unACKed: [152, 1433]
[12]1F34.0B50::2024/03/27-18:06:16.060566200 [Microsoft-Quic][C][TX][8] SH DestCid:c6f25eec KP:0 SB:1 (Payload 1291 bytes)
[12]1F34.0B50::2024/03/27-18:06:16.060568300 [Microsoft-Quic][C][TX][8]   STREAM ID:0 Offset:152 Len:1281
[12]1F34.0B50::2024/03/27-18:06:16.060568700 [Microsoft-Quic][pack][16492674416642] Encrypting
[12]1F34.0B50::2024/03/27-18:06:16.060574800 [Microsoft-Quic][pack][16492674416642] Finalizing

quic.zip

[nicely-dev]

Hi Nick, don't waste time on this yet, I found that if using text messages it works, so it seems to be an issue with sending binary data. Let me investigate more.

[nicely-dev]

I found my problem: the GUI was hogging the same thread as used by ClientStreamCallback. So nothing to do with the binary upper protocol. I hope this helps someone stuck with a delays in their QUIC streams.