lib: pixel: towards gstreamer/ffmpeg/opencv

[josuah]

This is a proposed implementation for:

• lib: pixel: towards gstreamer/ffmpeg/opencv/libcamera #86669

Downstream:

• drivers: video: sw_isp: software-based image processing #87007
• drivers: video: sw_stats: software-based statistics #87008
• drivers: video: sw_pipeline: glue lib/pixel and drivers/video #88839
• lib: pixel: distort: add distortion aka defisheye aka dewarp #88840

Example usage:

• samples/drivers/video/sw_pipeline/src/pipeline.c

Implementation notes:

• Functions named pixel_<a>_to_<b>() or pixel_<operation>_<a>(), where <a> and <b> are pixel formats name with frame/line/hist/avg suffix (no reliance on <zephyr/drivers/video.h> or <zephyr/drivers/display.h> formats)
• Heavy use of static inline: to allow the compiler to optimize the constants away and build operations that work on whole-lines without function call inside.
• No unit tests yet, as maybe using VIDEO_SW_GENERATOR to build the test data was interesting?

[josuah]

To test the samples on the native simulator:

west build -p -b native_sim samples/lib/pixel/print/ && west flash
west build -p -b native_sim samples/lib/pixel/resize/ && west flash
west build -p -b native_sim samples/lib/pixel/stats/ && west flash
west build -p -b native_sim samples/lib/pixel/stream/ && west flash

[nashif]

west build -p -b native_sim samples/lib/pixel/print/ && west flash

wow, this is nice

[josuah]

force-push:

• Switch resize function from memcpy() to memmove() to allow in-place resizing (thank you @IAmAnkur)
• Add missing input validation from video_sw_isp_set_signal (thank you @IAmAnkur)
• Try to address CI issues at the same time, after reproducing them locally

[josuah]

force-push:

• print the performance statistics on the stream sample
• switch samples from printf() to LOG_INF()
• Try to address CI issues at the same time, after reproducing them locally (different issues from last time)

[josuah]

force-push:

• Rebase on latest main (to go past Zephyr 4.1 and base other PRs on this)

[josuah]

Currently, the API is rather poor, not feature-rich like the gstreamer/opencv, very low-level.

For instance, one can imagine a python-style, data-centric, software-developer-friendly APIs like OpenCV:

/* A high-level abstraction of an image and associated operations  */
struct pixel_image *img = pixel_image(&video_dev);

/* Add processing steps, without performing them yet */
pixel_crop(img, 64, 64, PIXEL_HCROP_CENTERED | PIXEL_VCROP_CENTERED);
pixel_convert(img, PIXEL_FORMAT_RGB24);
pixel_denoise(img, PIXEL_DENOISE_LV5);

/* Actually perform all the conversion above */
uint8_t *output_buffer = pixel_process(img);
use(output_buffer);

Or it is possible to work with stream-oriented, stream-centric, system-integrator-friendly APIs like Gstreamer:

/* Instantiate every "tube" of the pipeline */
struct pixel_pipeline *pipeline;
struct pixel_pipe *source = pixel_new_pipe("videosource", "cam0");
struct pixel_pipe *crop = pixel_new_pipe("crop", "64x64", "middle");
struct pixel_pipe *torgb = pixel_new_pipe("torgb");
struct pixel_pipe *denoise = pixel_new_pipe("denoise", "lv5");
struct pixel_pipe *sink = pixel_new_pipe("buffersink", output_buffer);

/* Connect them together */
pixel_add_source(pipeline, source);
pixel_add_pipe(pipeline, crop);
pixel_add_pipe(pipeline, torgb);
pixel_add_pipe(pipeline, denoise);
pixel_add_sink(pipeline, sink);

/* Control the pipeline stream */
pixel_pipeline_ctrl(pipeline, PIXEL_STREAM_START);
pixel_pipeline_wait_sink(pipeline);
use(output_buffer);

Or maybe offload all of that to an upper layer project built on top, such as MicroPython, Arduino, Rust and 3rd-party modules. At the end of the day, each of these are doing the same thing.

In this PR, only the low-level plumbing is proposed, giving more time to discuss later about the porcelain part of this imaging kitchen sink.

[Copilot]

Pull Request Overview

This pull request implements a comprehensive pixel processing library with support for multiple pixel formats and operations such as streaming conversion, resizing, histogram generation, printing, and Bayer demosaicing. Key changes include:

• Addition of new pixel library source files (e.g., stream.c, stats.c, resize.c, print.c, formats.c, bayer.c) which implement various conversion and processing operations.
• Introduction of public API headers under include/zephyr/pixel/ that expose functions and macros for image processing pipelines.
• Updates to maintainer configuration to include the new Pixel library.

Reviewed Changes

Copilot reviewed 45 out of 51 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
lib/pixel/formats.c	New implementation of RGB / YUV conversion routines with a minor spelling typo in the header comment.
lib/pixel/bayer.c	Bayer demosaicing functions and stream converters; contains a typo in the header comment.
Other lib/pixel/*.c files	Implementation of pixel streaming, resizing, printing, and statistical functions.
include/zephyr/pixel/*.h files	Public API declarations and macros for the pixel processing library.
MAINTAINERS.yml	Updated to include the Pixel library with appropriate labels and tests.

Files not reviewed (6)

• lib/CMakeLists.txt: Language not supported
• lib/Kconfig: Language not supported
• lib/pixel/CMakeLists.txt: Language not supported
• lib/pixel/Kconfig: Language not supported
• samples/lib/lib.rst: Language not supported
• samples/lib/pixel/pixel.rst: Language not supported

Comments suppressed due to low confidence (2)

lib/pixel/formats.c:2

• Typo detected: 'Copyir' should be 'Copyright'.

 * Copyir (c) 2025 tinyVision.ai Inc.

lib/pixel/bayer.c:2

• Typo detected: 'Copyir' should be 'Copyright'.

 * Copyir (c) 2025 tinyVision.ai Inc.

[josuah]

Force-push:

• Fix typo in copyright statement.
• Add YUV24 <-> RGB24 functions implemented but missing from header.
• Add YUV24 <-> YUYV not implemented (only direct RGB24 <-> YUYV was there).
• Add tests for all YUV24 functions above.

[josuah]

It seems that we have a bit of time before reviews, so I went ahead and implemented the python-style API.

After all, if we can have an API that's easy like a script but is light even on microcontrollers and makes use of all hardware accelerators available, why shouldn't we have it... :)

struct pixel_image img;

pixel_image_from_buffer(&img, yuyvframe, sizeof(yuyvframe), 32, 8, VIDEO_PIX_FMT_YUYV);

pixel_image_convert(&img, VIDEO_PIX_FMT_RGB24);
pixel_image_kernel(&img, PIXEL_KERNEL_GAUSSIAN_BLUR, 5);
pixel_image_resize(&img, 120, 40);

pixel_image_to_buffer(&img, rgb24frame_out, sizeof(rgb24frame_out));
if (img.err != 0) {
        return img.err;
}

/* Print the image out for debug, send it to NPU/UVC/Bluetooth/WiFi/Ethernet/... */

It is only about time that ARM Helium SIMD instructions and custom 2.5D GPUs... are integrated

[josuah]

force-push:

• doc typo fix
• while at it, add missing Y8/GREY format support (useful for downstream PRs)
• add convenience functions to convert data from/to video buffers

[josuah]

Force-push:

• Add missing Y8/GREY support (used by downstream PR)
• Doc typo fixes
• Add wrappers to convert video buffers

[loicpoulain]

unfinished sentence, is it triggering the processing pipeline?

[josuah]

Good catch! I will check if I have more up to date documentation somewhere else and patch it.

[loicpoulain]

It will be added as latest operation, then maybe it should be clearer, naming it pixel_image_append_operation()?

[josuah]

Absolutely! 🙏

[loicpoulain]

why explicit inline of that function? same question for other functions

[josuah]

This is under-documented, but here is the idea.

pixel_rggb8_to_rgb24_3x3() is what performs the actual pixel operation of debayer:

static inline void pixel_rggb8_to_rgb24_3x3(const uint8_t rgr0[3], const uint8_t gbg1[3],
					    const uint8_t rgr2[3], uint8_t rgb24[3])
{
	rgb24[0] = ((uint16_t)rgr0[0] + rgr0[2] + rgr2[0] + rgr2[2]) / 4;
	rgb24[1] = ((uint16_t)rgr0[1] + gbg1[2] + gbg1[0] + rgr2[1]) / 4;
	rgb24[2] = gbg1[1];
}

This is called in a loop here:

__weak void pixel_line_rggb8_to_rgb24_3x3(const uint8_t *i0, const uint8_t *i1,
					     const uint8_t *i2, uint8_t *o0, uint16_t w)
{
	__ASSERT_NO_MSG(w >= 4 && w % 2 == 0);
	uint8_t il[3][3] = FOLD_L_3X3(i0, i1, i2);
	uint8_t ir[3][3] = FOLD_R_3X3(i0, i1, i2, w);

	pixel_grbg8_to_rgb24_3x3(il[0], il[1], il[2], &o0[0]);
	for (size_t i = 0, o = 3; i + 4 <= w; i += 2, o += 6) {
		pixel_rggb8_to_rgb24_3x3(&i0[i + 0], &i1[i + 0], &i2[i + 0], &o0[o + 0]);
		pixel_grbg8_to_rgb24_3x3(&i0[i + 1], &i1[i + 1], &i2[i + 1], &o0[o + 3]);
	}
	pixel_rggb8_to_rgb24_3x3(ir[0], ir[1], ir[2], &o0[w * 3 - 3]);
}

This builds a "tight loop" where as much execution as possible should be removed for best performance. Other implementations (i.e. here) actually implement debayer in just one big function or use static inline [here]

[josuah]

pixel_op_bayer_to_rgb24_2x2() you quoted is not so tight of a loop though, it is only called once per line, not per pixel... I feel like optimizing prematurely and I've not gotten time to do proper benchmarks.