[QUESTION] Framerate drop to 0

[bigbean17]

Framerate drop to 0 while bitrate is not 0 and internet connection is good.

** Logging **
This is the command that I'm using to send video feed from a camera to AWS, I can see the video on the AWS viewer by using below code:

UINT16 stringOutcome = snprintf(rtspPipeLineBuffer, RTSP_PIPELINE_MAX_CHAR_COUNT,
                                                    "rtspsrc location=%s ! rtph264depay ! h264parse ! avdec_h264 ! decodebin !"
                                                    "x264enc latency=0 bframes=0 speed-preset=very-fast bitrate=512 tune=zerolatency ! "
                                                    "appsink max-buffers=0 drop=false sync=false emit-signals=TRUE name=appsink-video ",
                                                    pSampleConfiguration->rtspUri);

Here is the logs

023-09-21 03:58:29.078 DEBUG   receiveLwsMessage(): Client received message of type: ICE_CANDIDATE
2023-09-21 03:58:29.078 VERBOSE lwsWssCallbackRoutine(): WSS callback with reason 10
2023-09-21 03:58:29.078 DEBUG   lwsWssCallbackRoutine(): Client is writable
2023-09-21 03:58:29.079 DEBUG   iceAgentLogNewCandidate(): New remote ice candidate discovered. Id: nrcT5LO+D. Ip: 3.27.90.140:61585. Type: relay. Protocol: udp.
2023-09-21 03:58:29.806 VERBOSE rtcpReportsCallback(): sender report no frames sent 1160073301
2023-09-21 03:58:29.806 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658319780667389 81372898070704 : 0 packets 0 bytes
2023-09-21 03:58:29.936 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658320338868397 81372898076943 : 0 packets 0 bytes
2023-09-21 03:58:30.006 VERBOSE rtcpReportsCallback(): sender report no frames sent 1160073301
2023-09-21 03:58:30.060 VERBOSE stepStateMachine(): State Machine - Current state: 0x0000000000000010, Next state: 0x0000000000000010, Current local state retry count [1], Max local state retry count [0], State transition wait time [0] ms
2023-09-21 03:58:30.137 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658321201036266 81372898086578 : 0 packets 0 bytes
2023-09-21 03:58:30.220 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658321559227949 237016 : 182 packets 23658 bytes
2023-09-21 03:58:30.298 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658321894106119 81372898094324 : 0 packets 0 bytes
2023-09-21 03:58:30.338 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658322068222805 247682 : 182 packets 23658 bytes
2023-09-21 03:58:30.438 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658322493743683 81372898101025 : 0 packets 0 bytes
2023-09-21 03:58:30.460 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658322589290814 258601 : 182 packets 23658 bytes
2023-09-21 03:58:30.572 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658323071116996 81372898107478 : 0 packets 0 bytes
2023-09-21 03:58:30.687 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658323565516265 279057 : 182 packets 23658 bytes
2023-09-21 03:58:30.867 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658324336665188 81372898121621 : 0 packets 0 bytes
2023-09-21 03:58:30.972 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658324787405255 304662 : 222 packets 49697 bytes
2023-09-21 03:58:30.986 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658324847490988 81372898127330 : 0 packets 0 bytes
2023-09-21 03:58:31.061 VERBOSE stepStateMachine(): State Machine - Current state: 0x0000000000000010, Next state: 0x0000000000000010, Current local state retry count [1], Max local state retry count [0], State transition wait time [0] ms
2023-09-21 03:58:31.107 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658325368490708 316838 : 261 packets 59088 bytes
2023-09-21 03:58:31.242 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658325947941925 81372898139629 : 0 packets 0 bytes
2023-09-21 03:58:31.397 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658326615196448 342963 : 365 packets 70382 bytes
2023-09-21 03:58:31.499 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658327054126644 81372898151991 : 0 packets 0 bytes
2023-09-21 03:58:31.645 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658327681801325 365313 : 456 packets 82864 bytes
2023-09-21 03:58:31.759 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658328168103722 81372898164441 : 0 packets 0 bytes
2023-09-21 03:58:31.817 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658328417806678 380736 : 521 packets 93205 bytes
2023-09-21 03:58:31.871 VERBOSE lwsWssCallbackRoutine(): WSS callback with reason 8
2023-09-21 03:58:31.871 ERROR   receiveLwsMessage(): operation returned status code: 0x5d000021
2023-09-21 03:58:31.871 WARN    signalingClientError(): Signaling client generated an error 0x5d000021 - '{"message": "Endpoint request timed out", "connectionId":"LlrUCfObSwMCIPg=", "requestId":"LlrXfGSkSwMFa5w="}'
2023-09-21 03:58:31.871 VERBOSE lwsWssCallbackRoutine(): WSS callback with reason 10
2023-09-21 03:58:31.871 DEBUG   lwsWssCallbackRoutine(): Client is writable
2023-09-21 03:58:32.025 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658329312053228 81372898177226 : 0 packets 0 bytes
2023-09-21 03:58:32.026 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658329317059872 399579 : 612 packets 114669 bytes
2023-09-21 03:58:32.061 VERBOSE stepStateMachine(): State Machine - Current state: 0x0000000000000010, Next state: 0x0000000000000010, Current local state retry count [1], Max local state retry count [0], State transition wait time [0] ms
2023-09-21 03:58:32.078 VERBOSE signalingClientGetCurrentState(): Signaling Client Get Current State
2023-09-21 03:58:32.238 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658330228414995 81372898187467 : 0 packets 0 bytes
2023-09-21 03:58:32.274 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658330383033389 421917 : 690 packets 131291 bytes
2023-09-21 03:58:32.389 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658330873823167 432201 : 716 packets 136902 bytes
2023-09-21 03:58:32.478 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658331256090718 81372898198952 : 0 packets 0 bytes
2023-09-21 03:58:32.571 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658331655072564 448572 : 729 packets 141314 bytes
2023-09-21 03:58:32.715 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658332276903513 81372898210361 : 0 packets 0 bytes
2023-09-21 03:58:32.821 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658332732253798 471144 : 729 packets 141314 bytes
2023-09-21 03:58:32.872 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658332947710832 81372898217857 : 0 packets 0 bytes
2023-09-21 03:58:33.021 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658333590950904 81372898225046 : 0 packets 0 bytes
2023-09-21 03:58:33.061 VERBOSE stepStateMachine(): State Machine - Current state: 0x0000000000000010, Next state: 0x0000000000000010, Current local state retry count [1], Max local state retry count [0], State transition wait time [0] ms
2023-09-21 03:58:33.064 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658333775007433 492995 : 729 packets 141314 bytes
2023-09-21 03:58:33.158 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658334179657347 81372898231625 : 0 packets 0 bytes
2023-09-21 03:58:33.208 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658334391386779 505911 : 729 packets 141314 bytes
2023-09-21 03:58:33.358 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658335036548420 519430 : 759 packets 163196 bytes
2023-09-21 03:58:33.375 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658335109155271 81372898242013 : 0 packets 0 bytes
2023-09-21 03:58:33.471 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658335523607160 529636 : 772 packets 169538 bytes
2023-09-21 03:58:33.582 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658335999372713 81372898251962 : 0 packets 0 bytes
2023-09-21 03:58:33.642 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658336255165933 544966 : 811 packets 183545 bytes
2023-09-21 03:58:33.776 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658336831523916 557043 : 837 packets 188805 bytes
2023-09-21 03:58:33.881 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658337284614618 81372898266326 : 0 packets 0 bytes
2023-09-21 03:58:33.892 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658337330619731 567502 : 850 packets 191078 bytes
2023-09-21 03:58:33.997 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658337782527600 576971 : 876 packets 195507 bytes
2023-09-21 03:58:34.061 VERBOSE stepStateMachine(): State Machine - Current state: 0x0000000000000010, Next state: 0x0000000000000010, Current local state retry count [1], Max local state retry count [0], State transition wait time [0] ms
2023-09-21 03:58:34.148 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658338432012555 590581 : 915 packets 200829 bytes
2023-09-21 03:58:34.166 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658338505301018 81372898279968 : 0 packets 0 bytes
2023-09-21 03:58:34.389 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658339463674007 612199 : 980 packets 210655 bytes
2023-09-21 03:58:34.461 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658339773168921 81372898294138 : 0 packets 0 bytes
2023-09-21 03:58:34.649 VERBOSE rtcpReportsCallback(): sender report 1160073301 16768658340581067302 635614 : 1045 packets 219035 bytes
2023-09-21 03:58:34.649 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658340581958937 81372898303177 : 0 packets 0 bytes
2023-09-21 03:58:34.781 VERBOSE rtcpReportsCallback(): sender report 936819997 16768658341149485607 81372898309519 : 0 packets 0 bytes
2023-09-21 03:58:34.852 VERBOSE lwsWssCallbackRoutine(): WSS callback with reason 10
2023-09-21 03:58:34.852 DEBUG   lwsWssCallbackRoutine(): Client is writable
2023-09-21 03:58:34.875 VERBOSE lwsWssCallbackRoutine(): WSS callback with reason 9

** Any design considerations/constraints **
I'm running the sdk on Jetson AGX and trying to reduce the latency as much as possible.

Thanks for helping~

[sirknightj]

Hi @bigbean17,

What are you using as viewer? Has a peer connection been established? Could you describe what sort of behavior are you seeing and after how long?

If you're using the JS SDK as viewer, you can use Chrome's webrtc tools (chrome://webrtc-internals ). For more information, see: https://docs.aws.amazon.com/kinesisvideostreams-webrtc-dg/latest/devguide/troubleshooting.html#determine-candidate

[bigbean17]

Hi @sirknightj

Viewer
I'm using the JS SDK and media playback viewer in AWS signalling channel to test, the issue exists on the both viewer.

Peer Connection
I believe the peer connection is established, as I can see videos from the viewers

Description
Approximately after playing the live video for 2-5 seconds, there may be a framerate drop to 0. And after 5-10 seconds, the video will continue playing, but sometimes it won't play anymore (framerate keeps 0 and I have to reconnect).

Posible Causes
The security camera I am using employs VBR, and I suspect this might be related. However, I tested it, and even when there are moving objects within the camera's field of view, the same issue occurs. When the framerate is zero, the bitrate is normal.

The camera is 360 degrees, and the issue is particularly severe when I change the camera's viewing position. For example, when the camera is facing forward, and I switch it to facing backward, there is a high probability that the framerate will drop to 0, and it takes at least 5 seconds before it resumes playback.

Thanks~

[disa6302]

@bigbean17 ,

Thank you for the details.

The camera is 360 degrees, and the issue is particularly severe when I change the camera's viewing position. For example, when the camera is facing forward, and I switch it to facing backward, there is a high probability that the framerate will drop to 0, and it takes at least 5 seconds before it resumes playback.

Is the camera switching happening in JS SDK or C SDK? If it is C SDK, how is it happening? Because a camera view switching would most likely require a SDP renegotiation in the application. Our SDKs internally do not support SDP renegotiation out of the box at the moment, and it would have to be an application specific implementation.