usb: device_next: new USB Video Class (UVC) implementation

[josuah]

Dependencies:

• drivers: video: introduce "GET" sub-operations #78603 replaced by something else to come
• drivers: video: allow allocation with a header preceding the buffer #79168 not required anymore
• video: emul: virtual driver for an imager and RX engine #79482
• drivers: video: Migrate ov7670 and smartdma to use video interfaces binding #88323
• drivers: video: CONFIG_VIDEO_SW_GENERATOR -> zephyr,video-sw-generator #88780
• drivers: video: controls: add the BASE and CAMERA controls #88924

Downstream:

• drivers: video: sw_pipeline: glue lib/pixel and drivers/video #88839

Build and run:

• usb: device_next: new USB Video Class (UVC) implementation #76798 (comment)

Try it on your own branch:

git remote add tvai https://github.com/tinyvision-ai-inc/zephyr
git fetch tvai
git cherry-pick tvai/pr-usb-uvc~1..tvai/pr-usb-uvc~0
west build -b <board> samples/subsys/usb/uvc/

[josuah]

I am grateful for the work on the USB and Video stacks of Zephyr, as well as the entire Zephyr tree, on the shoulder of which this is built.

[josuah]

Force push:

• fixes to make it compile with west build -b frdm_mcxn947/mcxn947/cpu0 (a board I happened to have, open to suggestions)
• added another commit (with I hope proper attributions!) to introduce fragments, which UVC and this implementation support.

[josuah]

Force-push:

• Add a sample to illustrate the usage. It is not yet tested but compiles.

[XenuIsWatching]

Where are these values coming from?

[josuah]

These are chosen arbitrarily, and I need to think about what should I do here: pick reasonable defaults? Deduce from other values? Let the user figure out by exposing a devicetree option?

[josuah]

In future work, it will be possible to ask the sensor for VIDEO_CID_PIXEL_RATE in combination with video_bits_per_pixel() to generate this field.

[XenuIsWatching]

I believe UVC can be BULK or ISO.... but I only see BULK supported here. Do you plan to have a way to select which type of EP?

[josuah]

I forgot to mention that only BULK is supported through this early version. I will need to spend more time investigating the device_next APIs for how ISO is implemented.

I cannot guarantee ETAs as I would do this feature on free time, but will add it to the roadmap: this will be needed at some point.

[josuah]

Very grateful for your reviews @XenuIsWatching I will force-push with the changes as soon as I get a chance to do so.

[josuah]

Force-push:

• control a selected source = <&dev>; device from the UVC stack directly
• wait that a format got selected before queuing video buffers to the USB
• use "discrete" frame intervals (FPS) values, which seemed to help getting it to work under Windows
• USB3CV compliant descriptors

[josuah]

force push: rebased on main

[josuah]

force push: changed the way descriptors are declared and introduce video controls at the descriptor level (no support for controls commands yet)

[josuah]

force push: implemented the UVC controls for the supported Video class controls.

I did not test this yet with the samples, but on the internal fork, we could get an IMX219 sensor with exposure and gain control from the host, format selection at startup (but not runtime, see [1]).

I believe that the last step for me is to test the sample on several boards that support device_next and fix the CI.

[josuah]

Known limitations:

[1]: there is currently no <video.h> API to let the application know that a format change occurred. For now, the sample waits that the host makes an initial choice, and does not reconsider its format selection.
A trade-off between supported feature and sample complexity.

[2]: there is currently no <video.h> API to let the video device communicate their min/max/step, so 0 is always picked as min, INT32/16/8_MAX as max, and 1 as step. Maybe the solution is to define an unit for each of the controls and have every sensor map this to their local definition. For instance, 1 µs for exposure, 1/1000 for gain, etc, and have the max value implicit. DONE

[3]: The "default" value is an arbitrary value of 1 instead of querying the controls of each sensor. DONE

[4]: There is no device const struct uvc_conf and only struct uvc_data, which wastes precious memory. DONE.

[5]: There are missing implementations for selector unit, extensions unit, encoding unit. TODO.

[6]: UVC introduces dependencies between some controls, i.e. auto-expopsure needs to be off for exposure to be accepted. TODO.

[7]: UVC has "error" control type reporting the last error. DONE

[8]: Only a single endpoint and output/streaming interface is supported per UVC interface. Composite devices (multiple UVC classes per device) are used instead for supporting multiple video streams per device. DONE

[9]: No documentation outside the devicetree bindings. DONE (no documentation needed anymore as no configuration required).

[10]: Supports custom header size, but not passing custom header data yet.

[11]: Still image capture (capturing one frame at full resolution) not supported.

[12]: USB3CV compliance tests not all passing since last rebase. TODO.

[13]: Announcing different resolutions/FPS for different connection speed not supported.

[14]: Asynchronous controls (the host setting a control, and a notification interrupt alert of completion) not supported.

Supported features:

[A]: Class API and enumeration

[B]: Custom control chains descriptors built from the devicetree layout Not supported by Linux

[C]: Selection of which controls to expose to the host via devicetree toggles Now controlled by the drivers.

[D]: Per-control entity tuning Sending all the controls requests to the first driver of the pipeline.

[E]: Handling of control commands end-to-end from host to Zephyr video device

[F]: Zephyr native Video API that allows to enqueue/dequeue frames like any other Zephyr video device.

[G]: Supports fragmented frames as discussed in #66994 and #72827

[H]: Can configure pixel format and resolution on the devicetree Done automatically by asking the video drivers.

[I]: Supports querying the min/max/current values of every controls end-to-end from host to Zephyr video driver.

[josuah]

Now that there is a video_query_ctrl(), it is possible to get the range for each control properly propagated from the driver all the way to the host over UVC. This time on FRDM-MCXN947 (with OV5640, on top of #89131)

$ v4l2-ctl --device /dev/video4 --list-ctrls

User Controls

                     brightness 0x00980900 (int)    : min=-15 max=15 step=1 default=0 value=-10
                       contrast 0x00980901 (int)    : min=0 max=255 step=1 default=0 value=0
                     saturation 0x00980902 (int)    : min=0 max=255 step=1 default=64 value=64

Still "in progress" as #88924 needs to be merged first... Or I can also implement just the minimum controls.

[josuah]

Force-push:

• For easier testing, rebased on top of #88780
• Added the same integration platforms as CDC ACM
• Simplified the data endpoint handling (split in multiple functions)

It is now possible to test without camera attached (again) through a snippet:

nRF52840 Dongle

west build --board nrf52840dongle --snippet video-sw-generator samples/subsys/usb/uvc/ \
    -DCONFIG_USB_DEVICE_STACK=n

FRDM-MCXN947

west build --board frdm_mcxn947/mcxn947/cpu0 --snippet video-sw-generator samples/subsys/usb/uvc/

Arduino Nicla Vision

west build --board arduino_nicla_vision/stm32h747xx/m7 --snippet video-sw-generator samples/subsys/usb/uvc/

RPi Pico

west build --board rpi_pico --snippet video-sw-generator samples/subsys/usb/uvc/

For each board above, this is the output:

[josuah]

force-push:

• rebased on latest main
• made the number of UVC buffers per instance configurable via Kconfig
• use USBD_MAX_BULK_MPS instead of hardcoded value

[josuah]

@jfischer-no it looks like the replacement of device by device_next is ongoing.

Does it make more sense to let the transition happen and merge UVC only after?

If so I will keep this PR as draft until the transition did happen, and update the branch to make sure it works on top of latest main, so that users can cherry-pick it UVC on top of their branch.

[josuah]

Force-push:

• Get UVC to work again on top of main

To try it on your branch:

git remote add tvai https://github.com/tinyvision-ai-inc/zephyr
git fetch tvai
git cherry-pick tvai/pr-usb-uvc~2..tvai/pr-usb-uvc~0

[sonarqubecloud]

Quality Gate failed

Failed conditions
E Reliability Rating on New Code (required ≥ C)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE