Low FPS in GStreamer aganst LiveCam

[aleksan8r-max]

I'm getting unexpectedly low FPS when using GStreamer.
I'm using image BananaPi_M2_zero_OV5640.img.
The ov5640camera initialization:
sudo media-ctl --device /dev/media1 --set-v4l2 '"ov5640 2-003c":0[fmt:YUYV8_2X8/640x480]'

When using the LiveCam script I don't see any problems with FPS. But for some reason it doesn't work with 1280x720 resolution. And outputs image to screen, but I need output to network.
cd software/python/LiveCam
python3 ./LiveCam.py
At the same time, the load on all 4 cores of the BananaPi M2 zero is approximately more than half.

When using GStreamer streaming to screen:
gst-launch-1.0 v4l2src device=/dev/video1 ! video/x-raw, width=640, height=480, framerate=30/1 ! videoconvert ! autovideosink
I get about 1 fps and lots of repeating warnings:

WARNING: from element /GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0/GstGLImageSinkBin:autovideosink0-actual-sink-glimage/GstGLImageSink:sink: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0/GstGLImageSinkBin:autovideosink0-actual-sink-glimage/GstGLImageSink:sink:
There may be a timestamping problem, or this computer is too slow.

At the same time, the load on all 4 BananaPi M2 zero cores is slightly higher than with LiveCam, but not 100%.

UDP streaming to another computer using GStreamer:
gst-launch-1.0 -v v4l2src device=/dev/video1 num-buffers=-1 ! video/x-raw, width=640, height=480, framerate=30/1 ! videoconvert ! jpegenc ! rtpjpegpay ! udpsink host=10.42.0.1 port=5200
Receiver:
gst-launch-1.0 -v udpsrc port=5200 ! application/x-rtp, media=video, clock-rate=90000, payload=96 ! rtpjpegdepay ! jpegdec ! videoconvert ! autovideosink
At the same time I get about 6 fps, only one core is loaded at 100%, the rest are around 0.

I got a BananaPi M2 zero hoping to get 30fps 1280x720 over the network, because you wrote:

we managed to transfer a 1280x720 UDP stream at 30 FPS using GStreamer.

What can be done with this? Or is the project dead?

[Qengineering]

Your media-ctl command should match the resolution you are using with GStreamer.
If you like to send 1280x720 with UTP, you must first initialise media-ctl with this command:

$ sudo media-ctl --device /dev/media1 --set-v4l2 '"ov5640 2-003c":0[fmt:YUYV8_2X8/1280x720]'

At the end of the line, you see the resolution (1280x720).
Now you run GStreamer with (assuming 192.168.178.84 is the IP address of the receiving device):

$ gst-launch-1.0 -v v4l2src device=/dev/video1 num-buffers=-1 ! video/x-raw, width=1280, height=720, framerate=30/1 ! videoconvert ! jpegenc ! rtpjpegpay ! udpsink host=192.168.178.84 port=5200

Please do note that the BananaPi has limited resources.
With 1280x720 @ 30 FPS, you utilise 25% CPU and 300 of the 500 MB RAM.

[aleksan8r-max]

Of course I tried 1280x720 resolution, and naturally the fps is even lower with it. But I didn't try to calculate how much exactly at 1280x720. I chose 640x480 just because the LiveCam script works with it. And naturally the media-ctl command matches to the resolution in GStreamer.
Thanks.

[Qengineering]

@aleksan8r-max, I owe you an apology. After some investigation, I reached the disappointing conclusion:
Compression is happening in jpegenc on the Banana Pi’s CPU. On a 1 GHz Cortex-A7 with 512 MB RAM, software JPEG encoding at 1280×720 is painfully slow — hence ~2 FPS on the receiving side.
Because I didn't get any warning or error on the Banana Pi's console, I initially thought the pipe was sound.
Sending the MJEP from the OV5640 fails due to memory issues (insufficient CMA).
I will remove the 30 FPS statement from the README.

[aleksan8r-max]

Maybe there is some solution without using JPEG encoding? LiveCam somehow works with the correct fps to the screen.

[Qengineering]

You can send the raw YUYV frames directly over UDP without JPEG encoding.
But the bandwidth is huge: raw YUYV at 1280×720 @ 30 FPS = 1280 × 720 × 2 bytes/pixel × 30 FPS ≈ 55 MB/s ≈ 440 Mbps
That is well above 100 Mb Ethernet and will choke the link.

You can always try:

Sender (Banana Pi Zero)
gst-launch-1.0 v4l2src device=/dev/video1
! video/x-raw,format=YUY2,width=640,height=480,framerate=30/1
! rtpvrawpay
! udpsink host=192.168.x.x port=5200

Receiver (PC)
gst-launch-1.0 udpsrc port=5200
! application/x-rtp,media=video,encoding-name=RAW,sampling=YCbCr-4:2:2,width=640,height=480,framerate=30/1
! rtpvrawdepay
! videoconvert
! autovideosink