[Question] kvsWebrtcClientMasterGstSample timestamp

[CarlosYeverino]

I noticed that kvsWebrtcClientMasterGstSample code on on_new_sample pulls appsink sample, gets gstreamer buffer and then it uses gstreamer buffer->pts (Presentation timestamp) to check only whether frame contains valid PTS

segment = gst_sample_get_segment(sample);
buf_pts = gst_segment_to_running_time(segment, GST_FORMAT_TIME, buffer->pts);
if (!GST_CLOCK_TIME_IS_VALID(buf_pts)) {
    DLOGE("[KVS GStreamer Master] Frame contains invalid PTS dropping the frame");
}

However, it doesn’t use it to pass timestamp to the frame data that is used in writeFrame method. Instead, it uses the streaming session video timestamp which is is an incremental value of constant SAMPLE_VIDEO_FRAME_DURATION:

    frame.presentationTs = pSampleStreamingSession->videoTimestamp;
    frame.decodingTs = frame.presentationTs;
    pSampleStreamingSession->videoTimestamp += SAMPLE_VIDEO_FRAME_DURATION; // assume video fps is 30
}
status = writeFrame(pRtcRtpTransceiver, &frame);

The writeFrame API is responsible for sending video frames over a WebRTC connection. It packetizes and sends media via the configuration specified by the RtcRtpTransceiver.
I noticed that writeFrame sets RTP packet timestamp to frame presentationTs:

rtpTimestamp = CONVERT_TIMESTAMP_TO_RTP(VIDEO_CLOCKRATE, pFrame->presentationTs);

QUESTIONS
1. Is there any reason no to set frame.presentationTs as frame.presentationTs = buffer->pts instead of incremental value of constant SAMPLE_VIDEO_FRAME_DURATION? It seems the current approach can be problematic in case of FPS mismatch or skipped frame on rtp reception.
2. In the scenario of kvsWebrtcClientMasterGstSample and AWS test Viewer page https://awslabs.github.io/amazon-kinesis-video-streams-webrtc-sdk-js/examples/index.html, who determines the FPS? The kvsWebrtcClientMasterGstSample application, isn't?
3. My understanding is that RTP packet timestamp is used only for synchronization between audio/video streams and the H.265/H.264 encoder timestamp is used for frame decoding/display order on Viewer side (AWS test Viewer page https://awslabs.github.io/amazon-kinesis-video-streams-webrtc-sdk-js/examples/index.html), can you please confirm this?

Thanks in advance.

[CarlosYeverino]

@stefankiesz @disa6302 @lherman-cs @suggestedfixes

[unicornss]

My understanding is that RTP packet timestamp is used only for synchronization between audio/video streams and the H.265/H.264 encoder timestamp is used for frame decoding/display order on Viewer side (AWS test Viewer page https://awslabs.github.io/amazon-kinesis-video-streams-webrtc-sdk-js/examples/index.html), can you please confirm this?

Yes, RTP timestamps are used for - Synchronizing audio and video streams, Managing packet arrival timing, Jitter buffer control.
Decoding and display order is primarily controlled by: incoming media's NAL unit headers, Frame dependencies (I, P, B frames). Sequence Parameter Set (SPS) and Picture Parameter Set (PPS).

[unicornss]

2. In the scenario of kvsWebrtcClientMasterGstSample and AWS test Viewer page https://awslabs.github.io/amazon-kinesis-video-streams-webrtc-sdk-js/examples/index.html, who determines the FPS? The kvsWebrtcClientMasterGstSample application, isn't?

Yes, WebRTC viewer acts as a passive consumer, cannot directly control or modify the FPS; it simply renders frames as they are received through WebRTC. Actual displayed FPS depends on- incoming stream's frame rate, network conditions, browser's rendering capabilities, available system resources in the viewer.

[unicornss]

Is there any reason no to set frame.presentationTs as frame.presentationTs = buffer->pts instead of incremental value of constant SAMPLE_VIDEO_FRAME_DURATION? It seems the current approach can be problematic in case of FPS mismatch or skipped frame on rtp reception.

In a typical media pipeline, especially when integrating with GStreamer, the buffer->pts value represents the actual presentation timestamp of the decoded/encoded frame. This frame timestamp reflects the actual timing of the media.

The use of an incremental SAMPLE_VIDEO_FRAME_DURATION in the SDK sample is intended for controlled demos and ensures predictable frame pacing. However, for production pipelines, setting frame.presentationTs = buffer->pts is preferable in real-time scenarios (assuming the encoder is stable enough to generate frames at regular intervals).

[sirknightj]

Note that sometimes pts from Gstreamer is not available (set to -1). In those cases, you would need to construct your own.

https://gstreamer.freedesktop.org/documentation/gstreamer/gstbuffer.html?gi-language=c#GstBuffer

[CarlosYeverino]

Hello @sirknightj
Based on the code I provided, the frame will be dropped in case of the invalid PTS and the setting of frame.presentationTs won't occur.

segment = gst_sample_get_segment(sample);
buf_pts = gst_segment_to_running_time(segment, GST_FORMAT_TIME, buffer->pts);
if (!GST_CLOCK_TIME_IS_VALID(buf_pts)) {
    DLOGE("[KVS GStreamer Master] Frame contains invalid PTS dropping the frame");
}