Question regarding multiple outputs and frame capture.

[kevinjacb]

Hello there, I have a question regarding multiple outputs from picamera. I am using the H264 encoder to both stream and record videos to files. Now I know that these 2 are handled in separate threads? I was wondering if it would be okay to also retrieve the frames by calling picam.capture_array() in the main loop? I tried this and sometimes it works fine, other times I get this error:
av_interleaved_write_frame which I believe is an error from ffmpeg. I am currently doing all this on rpi zero so I do understand that processing power is quite limited, but does all this cause issues with recording? In my case videos are getting recorded(when it works) but every 2 seconds it loses 4 or 5 frames. which means the video is not continuous. Any help will be greatly appreciated, thank you.

[davidplowman]

Hi, sorry I didn't spot the question sooner. If you could post a very minimal version of your code (barely more than a few lines, if possible...) that might be interesting.

I can't think of any reason why capturing an image while recording should cause a problem, though as you say a Pi Zero is almost certainly going to struggle. Presumably you're using the FfmpegOutput which just pipes stuff to an FFmpeg process, so it's not clear to me how that could go wrong.

[kevinjacb]

No problem! An example would be :

if __name__ == "__main__":

    picam = Picamera2()
    configuration = picam.create_video_configuration(main={"size":(1920, 1080), "format": "RGB888"}, lores={"size":(640,  480)})
    picam.configure(configuration)
    picam.controls.set_controls({"AfMode":controls.AfModeEnum.Continuous})


    encoder = H264Encoder()

    # stream output
    stream_output = FfmpegOutput('-f rtsp -rtsp_transport udp rtsp://192.168.0.123:1234/test')

    # record output
    rec_output = FfmpegOutput('-f mp4 ./test.mp4')

    #start streaming and recording
    encoder.output = [stream_output,rec_output]
    picam.start_encoder(encoder)
    
    while True:
        frame = picam.capture_array('main')
        # do something with frame ...

Seems like the loop isn't the issue, when I only stream this issue doesn't happen often. But when i record and stream, this error pops up after a few seconds. It may be due to the hardware limitations of the Pi Zero 2W.

The other question I have is this: #798
TLDR: How to handle this error without causing an exit.

Thank you!

[davidplowman]

I'm afraid I don't have a good answer for you. I think there's going to be some investigation required, and you may have to try a number of different things to see what does or doesn't cause problems. For example:

• Is it linked to the RTSP output, or do you get the same problem with another kind of network output (e.g. a vanilla udp/tcp stream)?
• I don't really understand ffmpeg and RTSP. Do you have some kind of RTSP server installed, and if so, what is it?
• Does it occur if the file output is a simple .h264 file?
• When the error occurs, has it written out the frame to the file? Can you see if it's truncated or corrupted?
• Do you have a more powerful device to try it on?

Sorry not to have anything better to try. Maybe someone else reading this might understand ffmpeg better.

[kevinjacb]

1. I've only tried rtsp so far, will check with the other streams
2. Yes I do have a mediamtx(previously simple-rtsp-server) running on another rpi in the same network for testing
3. If I remember right, it does, I initially used FileOutput to record the videos, but was faced with the issue of skipping frames(around 5) every 2 seconds, so I switched to ffmpegoutput for a continuous recording in mp4.
4. The program halts as the control is with self.ffmpeg.stdin.write() and since the error is not handled. Other threads work fine.
5. I do have a rpi 4 which hosts the rtsp server locally.

Thank you so much for taking the time to try and help me troubleshoot and resolve this issue :). I shall look for alternative solutions meanwhile.