drivers: video: switch from custom buffer queues to RTIO

[josuah]

• Video RTIO demo #92566 (PoC)
• Video: Rework buffer management #92370 (main effort)

Introduction

Switch ad-hoc video buffers and submission/completion queues (fifo_in/fifo_out fields) + polling API to RTIO.

Problem description

The video enqueue()/dequeue()/set_signal() operations use a custom struct for video buffers and FIFO storage. Furthermore, the polling API is rarely implemented and not used in the capture example.

Proposed change

Replace the submission/completion mechanism of Video by RTIO, benefiting from reduced context switching, reduced maintenance, and provides a polling API implicitly without user requiring to implement it.

Detailed RFC

Proposed change (Detailed)

At a high-level, the modifications would be:

• struct video_buffer -> rtio_cqe/rtio_sqe

• video_enqueue() -> rtio_sqe_acquire()+rtio_sqe_prep_read()+rtio_submit()

• video_dequeue() -> rtio_cqe_consume()

• (struct sensor_data).fifo_in -> (struct sensor_data).io_q

• (struct sensor_data).fifo_out -> (struct sensor_data).io_q (merged "I/O operation queue")

• (*video_api_enqueue_t)() -> (*submit)()

• k_fifo_put(&data->fifo_out, vbuf); -> rtio_iodev_sqe_ok(iodev_sqe, result);

• video_set_signal() -> not needed anymore

• k_poll_signal_raise(data->signal, VIDEO_BUF_DONE); -> not needed anymore (part of rtio_iodev_sqe_ok(iodev_sqe, result);)

• k_poll_signal_raise(data->signal, VIDEO_BUF_ERROR); -> rtio_iodev_sqe_err(iodev_sqe, result);

• vbuf->extra_fields -> Beginning of the RTIO buffer payload

Dependencies

All features needed are already in Zephyr, and the switch to RTIO is a matter of progressively converting the drivers.

A "RTIO shim" strategy is frequenty used as a glue layer to cover drivers not converted yet.

Concerns and Unresolved Questions

If the migration happens, sould struct video_buffer and several other APIs be kept for compatibility or a more convenient/flexible API be exposed instead?

Alternatives

Make use of struct net_buf instead of RTIO.

[josuah]

Cc: @loicpoulain @ngphibang @avolmat-st

[josuah]

Background information on integration with RTIO

• drivers: video: API enhancement #72959 (comment)
• RTIO: Add Workqueue support for delayed work items #77100
• rtio: API version 0.2.0, unstable #89853
• Thread on #rtio on the chat server

Background information on video buffer management:

• drivers: video: sw_generator: add more formats and try to simplify #85968 (comment)

[teburd]

I'm very supportive of this, so anything that needs to happen to help let me know

[josuah]

I did not put this in action yet (I plan to start this week), but some early feedback from discussions at Embedded Recipes:

• Gstreamer developer told me their only regret with V4L2 APIs was the unability to queue a buffer in multple queues at once. For instance encoding the same data at multiple resolutions simultaneously to dynamicallly switch depending on the network conditions.

• Linux video ecosystem has all sorts of facilities allowing to configure a pipeline and then let the read()/write() happen in the kernel. If there is already some kind of "splice(2)" equivalent for RTIO to connect a queue output to another input this can be convenient probably not a good idea as queueing is going to be customized by application a lot

For both it can be possible to work around on top of RTIO though!

Thank you for your support!