3 questions: (1) streaming YUV420 (2) turning off red and blue (3) adaptive callback

[ballen4705]

I'm working on code to analyse data from a scientific instrument, using an HQ camera on a RP 4B with a 64-bit OS. My code does the following:

• uses only main (not lowres or raw)
• format is main={"size": (2028, 1520),"format":"YUV420"}, interval 50ms (20 FPS)
• registers a pre_callback()
• starts a preview window

My callback (a) analyses data in the main array, then (b) modifies that array, annotating it with some text and lines. I can't use RGB. Only with YUV420 is the analysis fast enough to complete in ~40 ms.

I have three questions:

(1) I'd like to stream my modified frames so they can be viewed on a browser. The code in examples/mjpeg_server.py works well when main is formatted RBG, but fails on YUV420. Is there a way to stream YUV420? (The JpegEncoder() source code indicates that it can accept YUV420 input. Perhaps I have not figured out how to invoke it correctly?)

(2) My instrument uses only green light. Is there a simple way for me to "turn off" red and blue in the camera? For example setting black_level_b = black_level_r = 65535? Or setting the color correction matrix to CCM = diag(0,1,0)? I'd like the camera to set exposure/gains based entirely on the green pixels, and to leave the blue and red out of the luminance (Y) computation when creating the YUV420 data. (I've looked through the Raspberry Pi Camera Algorithm and Tuning Guide. Can I experiment with the settings by creating my own .json files? Is there a python call that can be used to make the image processing pipeline read this?)

[EDIT: reading the source code picamera2/picamera2.py I see Picamera2.load_tuning_file().]

(3) I am timing my callback code. I noticed that on startup, it appears that the callback is not invoked for every frame. Then, the call rate increases so that after a few seconds the handler is invoked for every frame. Is the callback initially invoked only on every n'th frame and timed, then gets invoked more frequently, once the caller is confident about the time to return? (Note: I get the mean callback rate by counting the total number of callbacks, then dividing by the elapsed wallclock time since the first callback.)

Thanks very much for assistance with any of these three issues!

Bruce

[davidplowman]

Hi, thanks for the questions. Let me try to answer.

1. I think you've posted this as a bug, and I've replied over on that thread. You may want to use the MJPEGEncoder instead.

2. You can set the colour gains, though unfortunately setting them to exactly zero is interpreted as meaning "turn the AWB on". But this should work

picam2.set_controls({'ColourGains': (0,001, 0.001)})

which should have the same effect. Those gains should be used in the AGC/AEC for calculating Y values, so I believe it will do what you want.

3. Can you say how you're measuring this? So far as I know, the pre_callback will be invoked on every frame we get from the camera, although if you're doing a lot of work in the callback (not recommend, normally), then you would get frame drops. Is that what you're noticing?

[ballen4705]

David,

thank you for the help.

(1) Got it. Others with this issue should look here: #565

(2) I'll try this and report back in due course. Some other ideas (thanks to Rolf Henkel for helpful discussions) are:

• first equation on page 38 of the guide "Raspberry Pi Control Algorithms", set the color correction matrix (CCM) to
  C = diag(0,1,0).
  This would mean that R_camera and B_camera no longer have any effect, so R_standard = B_standard = 0, and G_standard = G_camera.
• Alternatively I could also try

    | 0   1/3   0 |
C = | 0   1/3   0 |
    | 0   1/3   0 |

which would make R_standard = B_standard = G_standard = G_camera, again ignoring the R_camera and B_camera values. Note: if C needs to be invertible, then neither of these will work.

• set rpi.black_level_r = rpi.black_level_b = 65535
• making a ct_curve (perhaps just at color temp 4500) where the second and third values are both zero.

(3) I might be doing something stupid and don't want to waste your time. If I can reproduce the behaviour with a simple demo code, I'll post it here.

Cheers,
Bruce

[ballen4705]

Following through on point (3): it was my mistake.
Here is how I monitor the frequency (rate) at which the callback function is invoked:

# define a callback function called stamp_rate() ...
import cv2 as cv2
import numpy as np
import time

loops = -1
tstart = time.time()
def stamp_rate(request):
    global loops, tstart
    loops += 1
    if loops == 0:
        tstart = time.time()
    else:
        dt = time.time() - tstart
        rate = "calls/sec: " + str(np.round(loops/dt,2))
        with MappedArray(request, "main") as m:
            cv2.putText(m.array, rate, (90,90), cv2.FONT_HERSHEY_SIMPLEX, 3, (0,255,0), 6)

# ... then later in the code, attach this callback handler to the event loop
picam2.pre_callback = stamp_rate

This writes the average call/second rate onto each frame.

My mistake: I initialised tstart in the main body of the code, not at the first invocation of the callback handler. This generated a couple of seconds of offset, which made it appear that initially the callback handler was not being invoked often enough.

[ballen4705]

I wanted to follow through on item (2), which was to turn off the blue and red response. You were correct: the following worked perfectly!

picam2.set_controls({"ColourGains":(0.01,0.01)})

I have a new question, but will ask this in a new thread!