Correct gamma for stream::readPixels

[kpeeters]

On Android, in order to render the camera feed, I am using Stream with .stream(...) set to the camera external texture, and use this in the Texture::setStream. This works fine. I render this texture on a quad full-screen, and then call readPixels on the stream to get a bitmap of the full resolution camera image (as opposed to calling readPixels on the renderer).

If I now take that bitmap image and use it as a source for a new Texture, it seems that there is a brightness or gamma mismatch. In particular, if I make a camera shot of a dark room, the preview renders fine (the camera knows how to set the exposure or correct for low light), but once I stick the image obtained from Stream::readPixels into a new texture and render that, it is a lot darker. For a bright scene the preview and final bitmap are much closer or indistinguishable. I am using linear tonemapping.

What is the right way to compensate for this, so that the Stream::readPixels result gives what I actually see on the screen when I render this stream through a texture?

(I need a solution that works on Android versions down to 5.x, so I cannot use the more modern hardware buffer-based approach).

[romainguy]

That's because Filament materials are expected to output values with no transfer function applied ("in linear space"). When reading from a texture this is usually handled automatically by marking the texture as an sRGB texture so the sampler automatically applies the EOTF. Otherwise, you have to do it yourself in your material.

[romainguy]

BTW did you set a linear tone mapper or turn off post processing entirely? The latter will not properly encode the output for the display and the result will be too dark.

[kpeeters]

I use a linear tone mapper (create a ColorGrading object with .toneMapping(LINEAR) and use that in the View).

The texture I am using to display the stream has TextureFormat::SRGB8. I read this into a PixelBuffer with PixelDataFormat::RGBA (but that probably has nothing to do with it). I then load the resulting image data into another texture which also has TextureFormat::SRGB8. Shouldn't that work? Or does using a Stream for the texture data override that texture format somehow? I am happy to change something in the material to render the texture once I have read the pixel data, but how do I figure out which transformation to apply?

[romainguy]

External streams can't be configured as sRGB or any other format really. The format depends entirely on the underlying stream. However, if you are calling readPixels() on the Stream itself, that shouldn't matter since you are directly fetching the encoded data. I would double-check you are indeed using linear tone-mapping, because it might just be the culprit. Seeing code would be helpful.

[kpeeters]

I am absolutely positive I am using linear tone-mapping,

color_grading = filament::ColorGrading::Builder()
	.toneMapping(filament::ColorGrading::ToneMapping::LINEAR).build(*engine);
view->setPostProcessingEnabled(true);
view->setColorGrading(color_grading);

The texture for the camera preview is created using (all on the native side using the
C++ interface)

camera_texture = filament::Texture::Builder()
	.width(uint32_t(w))
	.height(uint32_t(h))
	.sampler(filament::backend::SamplerType::SAMPLER_EXTERNAL)
	.format(filament::backend::TextureFormat::SRGB8)
	.build(*engine);

I create a stream for the GL_TEXTURE_EXTERNAL_OES texture id (obtained
by calling GL functions directly on the Kotlin side and passing that id to my C++ code)
using

camera_stream = filament::Stream::Builder()
	.stream(camera_texture_id)
	.width(camera_width)
	.height(camera_height)
	.build(*engine);

camera_texture->setExternalStream(*engine, camera_stream);

This is rendered using an unlit material, all works fine.

To take a screenshot, I do

renderer-> beginFrame(...);
camera_stream->readPixels(0, 0, use_width, use_height, std::move(screenshot_buffer));
renderer->endFrame();

where buffer is a PixelBuffer with PixelDataFormat::RGBA. When the callback is triggered,
I create a new texture

image_texture = filament::Texture::Builder()
                   .width(camera_width)
                   .height(camera_height)
                   .sampler(filament::backend::SamplerType::SAMPLER_2D)
                   .format(filament::backend::TextureFormat::SRGB8_A8)		
                   .build(*filament_engine);

and feed it the pixel buffer just filled by readPixels,

image_texture->setImage(*engine, 0, std::move(screenshot_buffer));

Then render this again using an unlit material.

I would say that, assuming I haven't forgotten anything in the summary above, this
should work, and the object with the camera_texture should render the same as
the object with the image_texture. Would you agree? What I am seeing instead is
that for many scenes (especially very bright or dark ones) the second render (after
the screenshot) comes out a lot darker than the first (direct from the camera stream).

I was suspecting that this might be an issue in an automatic tone-mapping taking place
when rendering the texture-from-stream, which somehow isn't happening when I do
it later after the readPixels.

[romainguy]

There's nothing obviously wrong in your code, but you could try to remove .format(filament::backend::TextureFormat::SRGB8) to see what happens. I have no idea whether this is supposed to impact readPixels() (our implementation defers to glReadPixels(), but if that applies the EOTF that would explain the darkening).

[kpeeters]

It makes no difference, unfortunately; the image after the readPixels remains darkened.

[kpeeters]

I can get this to work right by doing an inverseTonemapSRGB in the material used to draw the camera stream texture, that is,

material.baseColor.rgb = inverseTonemapSRGB(material.baseColor.rgb);

In that case the rendered camera stream texture looks the same as the rendered texture which I generate from the data obtained from readPixels. Needless to say, the material to do that latter thing does not include the above inverseTonemapSRGB.

I can't say I understand why, because the Camera2 docs suggest that the camera texture is in sRGB format when the CaptureRequest.COLOR_CORRECTION_TRANSFORM key equals the unit matrix (which is the case for my device). Which would then imply that the camera texture format is the same as the format of the filament textures loaded from image data.

[romainguy]

I can't say I understand why, because the Camera2 docs suggest that the camera texture is in sRGB format when the CaptureRequest.COLOR_CORRECTION_TRANSFORM

That's not really relevant. There seem to be two problems then:

• Tone mapping is applied somewhere
• Your sRGB texture doesn't give you linear sRGB data after sampling

inverseTonemapSRGB() does two things:

• It decodes sRGB to linear sRGB
• It applies the inverse tone mapping curve