drivers: video: image statistics, libcamera-style APIs

[josuah]

Implementation:

• drivers: video: introduce an API for collecting statistics #85811
• drivers: video: sw_stats: software-based statistics #87008

Introduction

Cameras constantly adjust the colors to make image normal-looking in all conditions. To do this, Linux uses libcamera, which Zephyr lacks.

Problem description

There is no API for sending image statistics from the hardware back to the software.

By default, image sensors output is green and dark. Below the Zephyr kite under a bright spotlight, using default exposure level:

Some image sensors such as OV5640 come with built-in image correction so the image looks normal... at the cost of extra noise, so most sensors let Image Signal Processor (ISP) handle perform the color corrections.

There is no statistics format between the ISP hardware and the IPA software.

Proposed change

At first, focus on introducing APIs for enabling the application to implement the rest: image statistics reporting.

Detailed RFC

Proposed change (Detailed)

Implement a new API for collecting statistics, or reuse one of the existing for this purpose.

[ See below for updated version ]

This does not cover advanced features like "region of interest" (ROI, i.e. on a phone, tap on a corner of the display to focus there => needs API to ask the hardware to collect statistics for a smaller region of the display in particular).

Dependencies

• Implement video control framework #82158 (optional)

Concerns and Unresolved Questions

What API to use?

• The existing control API? (however, statistics are not )
• A new dedicated API?

What statistics format to use? libcamera defines sum_red, sum_green, sum_blue, histogram_brightness for the software back-end

Alternatives

• No standardized API, each hardware have their own APIs in custom headers.
• Re-use the Video Controls API for this with a custom struct

[josuah]

One alternative to having the R/G/B sum is to have a histogram for R, G, B as well, and define a bucket number of 1, which is giving the same value, except also supporting hardware reporting R, G, B histograms in the future if any.

This also allows to remove the bitmask, as a hardware not supporting any histogram of some type can set the number of buckets to 0. [EDIT: This does not work. 😖 with just one bucket we would know that all values are between min and max, not their sum.]

Example of complete ISP pipeline for desktop in Python:
https://github.com/cruxopen/openISP

[josuah]

After trying to implement something, it seems like some bitmap to negotiate the type of stats could be useful:

     Application                        Video device                          Hardware

          |                                   |                                   |
          | Bitmap of stats supported         |                                   |
          |---------------------------------->|                                   |
          |                                   | Fetches the stats from hardware   |
          |                                   | to (struct image_stats)           |
          |                                   |---------------------------------->|
          |                                   |<----------------------------------|
          | Bitmap of stats actually present  |                                   |
          | with just one bit set             |                                   |
          |<----------------------------------|                                   |
          |                                   |                                   |

This would permit the statistics to be self-describing (through the bitmap always as first field), and avoid too much ping-pong.

There can be functions that convert the stats from one format to another, and the hardware does not have to send the stats requested by the application: only preferences. This permits to implement all sorts of dynamic strategies (first ask for channel statistics, then ask for full R/G/B histogram if not supported) without pushing complexity in the drivers.

This should help with integrating together the "statistics zoo" that hardware might present, and allow very custom algorithms implemented with a generic API.

[josuah]

Example of intermediate "negotiation" in a situation where there is a lot of different statistics and a "good enough" Image Processing Algorithm (IPA) needs to be provided for a slightly mismatching API, without having to specialize the ISP to the hardware manually:

Application                 Video Device                 Library function
  |                              |                              |
  |                              |                              |
  | I want channel averages      |                              |
  |--(bitmask)------------------>|                              |
  |                              |                              |
  |                              |                              |
  |        here are RGB averages |                              |
  |<------------(stats+bitmask)--|                              |
  |                              |                              |
  |                              |                              |
  | I want Y channel average, here is what I received           |
  |--(stats+bitmask)------------------------------------------->|
  |                              |                              |
  |                              |                              |
  |            I computed the Y channel out of the RGB channels |
  |<-------------------------------------------(stats+bitmask)--|
  |                              |                              |
  |                              |                              |
  | Now I can compute the        |                              |
  | exposure tuning              |                              |
  |                              |                              |
  |                              |                              |
  | Adjust exposure to be 3000   |                              |
  |--(control)------------------>|                              |
  |                              |                              |

This is illustrated here in a basic, ad-hoc implementation:
https://github.com/tinyvision-ai-inc/tinyvision_zephyr_sdk/blob/659ab06ec5391830e456922f0109ca43e1b46602/samples/imx219_dual/src/main.c#L55-L66

Then, providing an ISP to a particular system becomes a matter of having either:

• The conversion functions between what the hardware has and what the IPA supports
• An IPA that immediately supports the statistics provided by the hardware

But not manually re-doing the IPA and adapt it to every new hardware, even optimized ones can be added incrementally and ported to other hardware (if it supports the same stats).