picamera2 with Gstreamer

[pablogarciaaunon]

Hi,

I would like to use picamera2 together with Gstreamer, using a Raspberry Pi Zero 2W and a Picamera Module V2, having in mind that:

• During the operation, I will need to change the resolution, crop the image, control exposure, take pictures, etc. It would be nice if those changes are as fast and smooth as possible (picamera2 definitely helps with this!).
• Video coded with H264.
• Using Gstreamer, mux KLVs into a MPEG transport stream, which contain different information regarding the position of the camera, timestamp, field of view, etc. This data changes over time and is fed into the stream using the element appsrc through the need_data callback.
• The final stream is packetized using RTP and sent via UDP.

I have been using the Gstreamer python bindings Gst, together with the raspicam plugin rpicamsrc. Since there is no such a plugin for picamera2, I was wondering if there is a way of doing something similar.

Things I have tried:

• Stream the camera video, already H264 compressed, similarly to the example capture_stream_udp.py, to the loopback address. Take that stream into the Gst pipeline, inject KLVs, packetize with RTP, and send it via UDP. It works "fine", but the video delay is higher and there is a reduction in the FPS (the video is clearly less smooth).
• Capturing the video frames into a numpy array and passing them to the Gst pipeline, using appsrc. For a Pi Zero 2, 720p, 40 fps, this is not viable.

Does anyone know how it could be done? Any clue?

I was wondering that maybe it could be done using file descriptors (somehow using FileOutput at the picamera2 side, and fdsrc at the Gst side??).

Thanks is advance

[davidplowman]

Hi, you don't mention libcamerasrc, which is the libcamera Gstreamer plugin. Might that help you? It's not developed by Raspberry Pi, but searching online for it should bring up some information.

[pablogarciaaunon]

Hello David,

thanks for your answer.

The problem with libcamerasrc, as far as I know, is that most of the options and tools available with picamera2 are not in that plugin (for instance, switching modes to save pictures rapidly or controlling image exposure on the fly). Moreover, I have faced problems related with the h264 compression using it (which I believe others have also faced); for example, these simple pipelines do not work (at least in my case):

gst-launch-1.0 libcamerasrc ! video/x-h264,width=640,height=480,framerate=30/1 ! fakesink
gst-launch-1.0 libcamerasrc ! video/x-raw,width=640,height=480,framerate=30/1 ! videoconvert ! v4l2h264enc ! fakesink

This does work though:

libcamera-vid --width 1280 --height 720 -t 0 -g 30 -b 8000000 -n --inline -o - | gst-launch-1.0 fdsrc fd=0 ! h264parse config-interval=-1 ! mpegtsmux ! rtpmp2tpay ! udpsink host=192.168.200.100 port=1234

But then you launch it as a command, not implementing in the Gst python library, losing then all control capabilities I would like to have...

Regards

[queue-bert]

@pablogarciaaunon Have you been able to find out a good solution? I'm using the RPi Camera Module 3 and I'm just trying to get a steady video stream using my RPi Zero 2 W but I haven't been able to do so over the network. I'm only able to get very low quality video to stream.

[pablogarciaaunon]

Hi.

The best solution I found is to send the compressed video using Picamera Python library, similar to this example, but using the loopback IP address. Now you can receive it in Gstreamer and then add the KLVs.

Regards

[tranhien1612]

@davidplowman @pablogarciaaunon Do you have an example of streaming video using gst in python?

[pablogarciaaunon]

Hi
I think I started with these examples

[tranhien1612]

@davidplowman @pablogarciaaunon I can stream video from pi camera via command line
libcamera-vid -t 0 --inline -o - | cvlc stream:///dev/stdin --sout '#rtp{sdp=rtsp://:8554/stream1}' :demux=h264
But when i use this command line, i don't control the pi camera (take photos, start and stop record video) by python scripts.
Because, i want to find an example about stream the video by python and i can see it via VLC app by rtsp://<ip>:<port>

[davidplowman]

Hi, I'm afraid I don't really know anything about gstreamer.

One of the problems with the libcamera gstreamer source is that it doesn't (cannot?) expose certain features of the camera system. For example, it may be difficult to get multiple version of the image and different resolution and in different formats, and use them for different purposes. Dynamic control of the camera may also be more difficult - as I said, I don't really know too much.

The loopback idea suggested above sounds interesting, so that might be another thing to try.

As you say, cvlc enables RTSP streaming. Possibly you could get Picamera2 to write to a named pipe (instead of a regular file) and then use cvlc to read from the pipe. You could probably put the pipe in /dev/shm (i.e. keep the file in memory, and not actually write to disk). It feels to me like that should work, so if I were doing this I might try that first.

[pablogarciaaunon]

@tranhien1612 As I mentioned, the approach I took is to use the Picamera2 library to stream the video to the loopback address (see this example), having control over the camera, and then you can use gstreamer to receive and manipulate as you want (either using Gstreamer in terminal or with its Python bindings).

I also tried using a fifo instead the loopback address, but I didn't see any improvement nor advantage.