drivers: video: sw_pipeline: glue lib/pixel and drivers/video #88839

josuah · 2025-04-21T00:59:47Z

Dependencies:

Sibling:

drivers: video: sw_isp: software-based image processing #87007

This allows some very short applications be written to see a video stream from their host as a webcam

#define WIDTH_IN 64
#define HEIGHT_IN 64

#define WIDTH_OUT 128
#define HEIGHT_OUT 32

void app_tune_rgb24line(const uint8_t *rgb24in, uint8_t *rgb24out, uint16_t width)
{
        for (size_t w = 0; w < width; w++) {
                rgb24out[w * 3 + 0] = CLAMP((-64 + rgb24in[w * 3 + 0]) * 2, 0x00, 0xff);
                rgb24out[w * 3 + 1] = CLAMP((-64 + rgb24in[w * 3 + 1]) * 1, 0x00, 0xff);
                rgb24out[w * 3 + 2] = CLAMP((-64 + rgb24in[w * 3 + 2]) * 2, 0x00, 0xff);
        }
}

PIXEL_RGB565LESTREAM_TO_RGB24STREAM(step_rgb565_to_rgb24, WIDTH_IN, HEIGHT_IN);
PIXEL_SUBSAMPLE_RGB24STREAM(step_resize_rgb24, WIDTH_IN, HEIGHT_IN);
PIXEL_RGB24LINE_DEFINE(step_tune_rgb24, app_tune_rgb24line, WIDTH_OUT, HEIGHT_OUT);
PIXEL_RGB24STREAM_TO_YUYVSTREAM_BT709(step_rgb24_to_yuyv, WIDTH_OUT, HEIGHT_OUT);

int app_init_pipeline(void)
{
        const struct device *dev = DEVICE_DT_GET(DT_NODELABEL(video_sw_pipeline));
        struct pixel_stream *strm;

        strm = pixel_stream(&step_rgb565_to_rgb24, &step_resize_rgb24, &step_tune_rgb24,
                            &step_rgb24_to_yuyv, NULL);
        video_sw_pipeline_set_pipeline(dev, strm, VIDEO_PIX_FMT_RGB565, WIDTH_IN, HEIGHT_IN,
                                       VIDEO_PIX_FMT_YUYV, WIDTH_OUT, HEIGHT_OUT);
        return 0;
}

SYS_INIT(app_init_pipeline, POST_KERNEL, CONFIG_KERNEL_INIT_PRIORITY_DEVICE);

Input from the webcam (after commenting out &step_resize_rgb24, &step_tune_rgb24, then WIDTH/HEIGHT_OUT=64):

Input from the webcam (after commenting out &step_tune_rgb24):

Output from the webcam (nothing commented):

josuah · 2025-04-21T01:00:07Z

The logs give performance statistics for each block of the pipeline:

[00:00:49.621,000] <inf> pixel_stream:   638 us on [step_rgb565_to_rgb24 (pixel_rgb565lestream_to_rgb24stream) ((64))x((64))]
[00:00:49.621,000] <inf> pixel_stream:  1316 us on [step_rgb24_to_yuyv (pixel_rgb24stream_to_yuyvstream_bt709) ((64))x((64))]
[00:00:49.621,000] <inf> pixel_stream:     0 us on [export video_sw_pipeline]

josuah · 2025-04-21T01:04:08Z

Nothing specific to UVC though: network capture or a display also work for instance.

This is a video device for connecting to any input/output and do some processing in-between.

In main.c there is just 2 lines specific to UVC.

josuah · 2025-04-21T01:07:34Z

Converted to draft until the dependencies are merged :)

Copilot

Pull Request Overview

This PR introduces a comprehensive set of changes to integrate and extend the pixel library and associated video driver components, enabling a new software video pipeline and generator implementation. Key changes include:

New header files and implementations for pixel stream processing, resizing, color conversions, kernel filtering, and bayer pattern processing.
Addition of device driver support for video software pipelines and generators with updated DTS bindings.
Update of documentation and maintainers metadata for the pixel library.

Reviewed Changes

Copilot reviewed 82 out of 87 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
include/zephyr/usb/class/usbd_uvc.h	Added UVC header but with DFU documentation that needs updating.
include/zephyr/pixel/stats.h	New statistics functions with a spelling error in channel naming.
include/zephyr/pixel/resize.h	Added resizing functions with a minor typographical error.
include/zephyr/pixel/kernel.h	Added kernel filter definitions with a minor grammatical error.
drivers/video/video_sw_pipeline.c	Introduced video pipeline thread and buffer handling logic.
drivers/video/video_sw_generator.c	Updated generator driver with new macro-based device definition.
dts/bindings/*	New DTS bindings for video pipeline, generator, and UVC device.
MAINTAINERS.yml	Updated metadata for the pixel library.

Files not reviewed (5)

drivers/video/CMakeLists.txt: Language not supported
drivers/video/Kconfig: Language not supported
drivers/video/Kconfig.sw_generator: Language not supported
drivers/video/Kconfig.sw_pipeline: Language not supported
lib/CMakeLists.txt: Language not supported

include/zephyr/usb/class/usbd_uvc.h

include/zephyr/pixel/stats.h

include/zephyr/pixel/resize.h

include/zephyr/pixel/kernel.h

nordicjm · 2025-04-22T06:37:47Z

drivers/video/Kconfig.sw_pipeline

+config VIDEO_SW_PIPELINE_THREAD_PRIORITY
+	int "Video Software Pipeline thread priority"
+	default 2
+	help


depends on VIDEO_SW_PIPELINE

nordicjm · 2025-04-22T06:38:59Z

lib/pixel/CMakeLists.txt

+zephyr_library_sources(bayer.c)
+zephyr_library_sources(formats.c)
+zephyr_library_sources(kernel.c)
+zephyr_library_sources(print.c)


shouldn't this only be included if the Kconfig is set?

AFAIU it is already done from the parent CMakeLists.txthttps://github.com/zephyrproject-rtos/zephyr/pull/88839/files#diff-4803677bd280f4dded52c428e3c8ac6c618508dcabcbe880b68be85897c2ad41:

add_subdirectory_ifdef(CONFIG_PIXEL pixel)

Let me know if there should also be a _ifdef condition on the inside.

github shows 4 lines but was only meaning the print.c line since there's a Kconfig for selecting that

CONFIG_PIXEL_PRINT_NONE=y allows to keep all the pixel_print_ debug statements which help for troubleshooting the tests and useful in samples, but turn them off the rest of the time.
So when it is on, the pixel.c functions are still called but do nothing.
Bad idea?

nordicjm · 2025-04-22T06:39:57Z

samples/drivers/video/sw_pipeline/CMakeLists.txt

+target_sources(app PRIVATE src/main.c)
+target_sources(app PRIVATE src/pipeline.c)


Suggested change

target_sources(app PRIVATE src/main.c)

target_sources(app PRIVATE src/pipeline.c)

target_sources(app PRIVATE src/main.c src/pipeline.c)

Good to know about this, I will preserve this style in future PRs.

nordicjm · 2025-04-22T06:40:11Z

samples/drivers/video/sw_pipeline/CMakeLists.txt

+
+cmake_minimum_required(VERSION 3.20.0)
+find_package(Zephyr REQUIRED HINTS $ENV{ZEPHYR_BASE})
+project(usb_video)


should match name of sample

I must have forgotten to change it. Adjusted thank you.

By having an "auto-fallback" mechanism, the VIDEO_SW_GENERATOR did shadow the fact that "platform:mimxrt1064_evk:SHIELD=dvp_fpc24_mt9m114" was missing the full specification, and therefore was not matched at all: the shield was not selected, but the CI still worked. Signed-off-by: Josuah Demangeon <me@josuah.net>

Make the video software generator a devicetree node, which allows enabling several instances, and select it as chosen { zephyr,camera = &... }; node. It can be enabled via a `video-sw-generator` snippet. Signed-off-by: Josuah Demangeon <me@josuah.net>

Convert the log module declarationto the new syntax introduced in 5e34681 proposing a more compact syntax. This fixes the log level not being updated in the "capture" sample. Signed-off-by: Josuah Demangeon <me@josuah.net>

Variables that holds pointers to 'const struct device' typically end-up with '_dev'. The tcpserversink video sample did not have it. Add it. Signed-off-by: Josuah Demangeon <me@josuah.net>

The zephyr,camera was used in several samples, but was not documented. Add the missing documentation entry for it in the list. Signed-off-by: Josuah Demangeon <me@josuah.net>

Add all the base controls present like they are in Linux into Zephyr, limited to those that can apply in the current system: - Buttons are left as integer for now. - Some description is modified to fit the Zephyr situation. - For the minimum number of buffer, Zephyr uses a different mechanism. - No audio support through the video subsystem. - Homogenize the wording Signed-off-by: Josuah Demangeon <me@josuah.net>

Introduce a new USB Video Class (UVC) implementation from scratch. It exposes a native Zephyr Video driver interface, allowing to call the video_enqueue()/video_dequeue() interface. It will query the attached video device to learn about the pipeline capabilities, and use this to configure the USB descriptors. At runtime, this UVC implementation will send this device all the control requests, which it can then dispatch to the rest of the pipeline. The application can poll the format currently selected by the host, but will not be alerted when the host configures a new format, as there is no video.h API for it yet. Signed-off-by: Josuah Demangeon <me@josuah.net>

The Arduino Nicla Vision board supports the new device_next USB stack and can be used for testing USB Device features such as UVC. Signed-off-by: Josuah Demangeon <me@josuah.net>

Following the addition of USB Video Class, this adds a sample that makes use of the &zephyr,camera chosen node of any board to stream the video source to the host. A fallback video-emul.overlay is provided for test and debugging purpose for devices without a camera. Signed-off-by: Josuah Demangeon <me@josuah.net>

josuah · 2025-04-28T00:14:12Z

Force-push:

Rebase on latest UVC branch

Note, this might crrash due to a bug in the video-sw-generator fixed in #85968 (not merged yet).

In order to test it, you will need to specify a framerate manually, such as ffplay -framerate 30 /dev/video4

The "pixel" library aims facilitating the implementation of all image processing tasks, such as pixel conversion, with a focus on low-resource environments. The processing functions scale down to per-pixel "kernels" to line-based conversion to full streams of several frames. Signed-off-by: Josuah Demangeon <me@josuah.net>

The newly introduced lib/pixel features utilities that help composing video pipelines together for the purpose of stream processing, as well as debug utilities. Signed-off-by: Josuah Demangeon <me@josuah.net>

Introduce a connector between the lib/pixel API to the drivers/video API. An function loads a lib/pixel stream into this new driver, then the driver will load the frames in and out of this stream. Signed-off-by: Josuah Demangeon <me@josuah.net>

josuah · 2025-05-01T13:01:07Z

A driver is not a good place for the application:

In firmware the barrier between driver and application is thin, but good to have still:

Grateful for the reviews, applied to the upstream PR.

github-actions bot added area: Process area: Video Video subsystem area: USB Universal Serial Bus area: Samples Samples area: Build System area: Devicetree labels Apr 21, 2025

github-actions bot requested review from 57300, decsny, galak, jeremybettis, jfischer-no, kartben, loicpoulain, MaureenHelm, nashif, ngphibang, nordicjm, rruuaanng, stephanosio, tejlmand and tmon-nordic April 21, 2025 01:00

github-actions bot assigned kartben Apr 21, 2025

josuah marked this pull request as draft April 21, 2025 01:07

josuah force-pushed the pr-video-sw-pipeline branch from 70bee3f to ff6694a Compare April 21, 2025 01:10

josuah mentioned this pull request Apr 21, 2025

lib: pixel: towards gstreamer/ffmpeg/opencv #86671

Open

kartben requested review from Copilot and removed request for stephanosio April 21, 2025 03:33

Copilot AI reviewed Apr 21, 2025

View reviewed changes

include/zephyr/usb/class/usbd_uvc.h Show resolved Hide resolved

include/zephyr/pixel/stats.h Outdated Show resolved Hide resolved

include/zephyr/pixel/resize.h Outdated Show resolved Hide resolved

include/zephyr/pixel/kernel.h Outdated Show resolved Hide resolved

josuah mentioned this pull request Apr 21, 2025

drivers: video: CONFIG_VIDEO_SW_GENERATOR -> zephyr,video-sw-generator #88780

Open

nordicjm reviewed Apr 22, 2025

View reviewed changes

josuah force-pushed the pr-video-sw-pipeline branch from ff6694a to 9881636 Compare April 22, 2025 17:49

josuah added 9 commits April 27, 2025 11:59

samples: drivers: video: tcpserversink: add _dev suffix to variable

4082b1d

Variables that holds pointers to 'const struct device' typically end-up with '_dev'. The tcpserversink video sample did not have it. Add it. Signed-off-by: Josuah Demangeon <me@josuah.net>

doc: build: dts: api: document the zephyr,camera chosen node

a7f4a6a

The zephyr,camera was used in several samples, but was not documented. Add the missing documentation entry for it in the list. Signed-off-by: Josuah Demangeon <me@josuah.net>

boards: nicla_vision: add usbd test feature

4791b31

The Arduino Nicla Vision board supports the new device_next USB stack and can be used for testing USB Device features such as UVC. Signed-off-by: Josuah Demangeon <me@josuah.net>

josuah force-pushed the pr-video-sw-pipeline branch from 9881636 to adf0a85 Compare April 28, 2025 00:10

josuah mentioned this pull request Apr 28, 2025

drivers: video: sw_isp: software-based image processing #87007

Draft

josuah force-pushed the pr-video-sw-pipeline branch from adf0a85 to 4ac3514 Compare April 28, 2025 00:26

josuah mentioned this pull request Apr 28, 2025

usb: device_next: new USB Video Class (UVC) implementation #76798

Draft

josuah force-pushed the pr-video-sw-pipeline branch from 4ac3514 to 728155d Compare April 28, 2025 00:37

josuah mentioned this pull request Apr 28, 2025

drivers: video: forward packets between devices #89190

Open

josuah added 3 commits April 28, 2025 13:47

samples: lib: pixel: show usage of the pixel library

dde0bf2

The newly introduced lib/pixel features utilities that help composing video pipelines together for the purpose of stream processing, as well as debug utilities. Signed-off-by: Josuah Demangeon <me@josuah.net>

josuah force-pushed the pr-video-sw-pipeline branch from 728155d to fadb076 Compare April 28, 2025 13:56

josuah closed this May 1, 2025

		target_sources(app PRIVATE src/main.c)
		target_sources(app PRIVATE src/pipeline.c)

	target_sources(app PRIVATE src/main.c)
	target_sources(app PRIVATE src/pipeline.c)
	target_sources(app PRIVATE src/main.c src/pipeline.c)

drivers: video: sw_pipeline: glue lib/pixel and drivers/video #88839

drivers: video: sw_pipeline: glue lib/pixel and drivers/video #88839

Uh oh!

Conversation

josuah commented Apr 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

josuah commented Apr 21, 2025

Uh oh!

josuah commented Apr 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

josuah commented Apr 21, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

josuah commented Apr 28, 2025

Uh oh!

josuah commented May 1, 2025

Uh oh!

Uh oh!

josuah commented Apr 21, 2025 •

edited

Loading

josuah commented Apr 21, 2025 •

edited

Loading