Applying any non-zero DistortionRGB parameters has no visible effect on D435 (misalignment between RGB and IR textures)

[Zauzolkov]

Required Info
Camera Model	D435
Firmware Version	5.16.0.1
Operating System & Version	GNU/Linux (Ubuntu 22.04 and NixOS)
Kernel Version (Linux Only)	6.6.76
Platform	PC
SDK Version	2.56.2
Language	Not applicable
Segment	Underwater Robotics

Issue Description

There is no observable effect on image from RGB camera after applying custom calibration parameters with DistortionRGB values filled.

Actions performed

1. Executed CustomCalibration example to calculate distortion parameters (DynamicCalibrationAPI/2.14.2.0/examples/CustomCalibration), so I got XML file after this step.
2. Executed CustomRW tool (from DEB package) to write new calibration parameters to device:

   $ Intel.Realsense.CustomRW  -w -rgb -f CalibrationParameters.xml 
   CustomRW for Intel RealSense D400, Version: 2.13.1.0
   
     >>> TOO LONG LOG CROPPED <<<
   
     HasRGB: 1
   
     resolutionRGB: 1920 1080
   
     FocalLengthColor: 2413.610107 2413.610107
     PrincipalPointColor: 995.437988 774.578003
     DistortionColor: 1.084310 -1.286550 0.091234 0.003310 0.502163
     RotationLeftColor: 0.999957 0.009198 0.000679
                       -0.009133 0.997752 -0.066389
                       -0.001289 0.066379 0.997794
     TranslationLeftColor: 12.566200 -25.375401 195.740997
   

3. Started video streams in realsense-viewer to observe images from cameras.

Observed results

The image from the RGB camera does not change, while the IR cameras demonstrate different profiles of optical distortion (depending on applied configuration).

There is demonstration of same static scene captured by D435 with different calibration configurations:

Factory Calib	Custom Calib

As you can see, RGB image remains unchanged.

Please ignore the low quality of the depth map on "Custom Calib" screenshot as this configuration is intended for underwater conditions.

DistortionRGB values from XML files

---

Factory:

<param name = "DistortionRGB">
    <value>0</value>
    <value>0</value>
    <value>0</value>
    <value>0</value>
    <value>0</value>
</param>

---

Custom:

<param name = "DistortionRGB">
    <value>1.08431</value>
    <value>-1.28655</value>
    <value>0.0912345</value>
    <value>0.00331035</value>
    <value>0.502163</value>
</param>

---

Intrinsic parameters read from D435

These values were obtained using the rs-enumerate-devices -c, so calibration parameters are stored on device (confirmation that write operation was actually performed by CustomCW).

 Intrinsic of "Color" / 640x480 / {YUYV/RGB8/BGR8/RGBA8/BGRA8/Y8}
  Width:        640
  Height:       480
  PPX:          335.750244140625
  PPY:          344.256866455078
  Fx:           1072.71557617188
  Fy:           1072.71557617188
  Distortion:   Inverse Brown Conrady
  Coeffs:       2.08431005477905        -2.28655004501343       2.0912344455719         2.00331044197083  2.50216293334961
  FOV (deg):    33.21 x 25

Expected results

I expected that the RGB camera image would change (after writing the distortion parameters) in a similar way as IR cameras does. But should it?

What have I tried to resolve this problem

• Turned video streams off and on (in realsense-viewer), since this is necessary to observe changes after writing custom parameters via CustomRW.
• Performed hardware reset and USB re-plugging.
• Performed factory reset of calibration settings (using both "Camera Calibration" menu in realsense-viewer and Intel.Realsense.CustomRW -g).
• Upgraded firmware to latest available version.
• Tried to write parameters while video streams are turned off (disabled in realsense-viewer).
• Executed CustomRW with and without -rgb / --force-rgb argument.
• Tried to manually set bigger values in DistortionRGB section of XML file, but there are still no any observable changes in image.

Context

Here is some additional context, in case I am approaching the problem incorrectly — i.e., a possible XY problem regarding this issue.

Why do we try to perform custom calibration

There is known problem of distorted (curved) depth map that occur in underwater conditions. The custom calibration procedure resolved this problem: planar objects now remain flat in the depth map thanks to proper undistortion of the depth images.

Why do we try to apply RGB camera calibration parameters

We noticed some alignment issues: while the 3D mesh is textured correctly using IR images, there is a serious mismatch when RGB Color Stream is selected in the Texture Source menu. We used realsense-viewer tool to check 3D mesh texturing, so our custom (client) software is not causing this issue itself (as it reproduced with tool from official Intel SDK).

Example of misalignment

[MartyG-RealSense]

Hi @Zauzolkov I see that you have read the extensive technical discussion on calibrating the camera underwater at IntelRealSense/realsense-ros#1723

It is such a long discussion that some of it is hidden from view, so please left-click on the 76 hidden items - load more link at IntelRealSense/realsense-ros#1723 (comment) to see the full conversation.

In this discussion, one of my Intel RealSense colleagues who is a calibration expert provides very detailed advice, including advice at IntelRealSense/realsense-ros#1723 (comment) about the kind of manual custom calibration that you have been attempting.

Their advice covers custom depth calibration though, and they had not attempted doing so with RGB calibration values in as much detail, though they thought that the same principles as the depth calibration procedure would logically be applicable to RGB.

Have you already read this advice about custom calibration, please?

[Zauzolkov]

@MartyG-RealSense Thank you for your reply. I will try to follow your recommendations as soon as possible (when I will have the device in my hands again tomorrow) and will try to manually change the RGB intrinsics in that way as suggested in the mentioned message. I haven't followed this advice yet, because the quality of distortion correction for IR cameras is already quite good enough after performing Custom Calibration procedure. Of course, I will try to apply these instructions to the RGB camera intrinsic parameters. Thanks for this advice.

However, at the moment the problem is not so much about finding/calculating the appropriate parameters, but the fact that the device doesn't seem to respond to any changes of the DistortionRGB parameters. I will certainly try to write (upload) such coefficients according to the mentioned recommendation. But I'm afraid that I will not see any effect again, because I have already tried to manually set various coefficients, but did not observe any visible changes in the RGB camera stream.

I will share the results after trying to set the parameters in the mentioned way. However, maybe there are some other ideas or recommendations regarding the problem that changing the RGB camera intrinsic parameters does not lead to any visible changes in the video stream? Or maybe there are some other recommendations for correcting the texture alignment in my situation?

Thank you very much for your attention to my problem!

[MartyG-RealSense]

Have you tried performing a calibration with the default unmodified graphical-interface Dynamic Calibration tool, which calibrates depth and RGB in the same session, to see whether the RGB values have changed after the completed calibration has been saved to the camera hardware?

In your kindly provided images of calibration, it looks as though the recommended target images are not being used for the appropriate calibration tool. The Dynamic Calibration tool uses a target image that is a series of vertical bars rather than a traditional checkerboard.

https://dev.intelrealsense.com/docs/dynamic-calibration-print-target

---

Whilst the On-Chip depth calibration method in the RealSense Viewer (which does not calibrate RGB) uses a noise pattern.

https://dev.intelrealsense.com/docs/self-calibration-for-depth-cameras#appendix-a-example-textured-target

---

The four-dot calibration image in your Examples of misalignment section is not used for calibrating depth quality with On-Chip calibration, but instead with the RealSense Viewer's Focal Length calibration (which helps to reduce distortions) or the Tare calibration (which helps to improve depth measurement accuracy).

The Viewer's On-Chip, Tare and Focal Length calibration methods do not calibrate RGB. Only the Dynamic Calibration tool (or the custom calibration API version of it) can calibrate both depth and RGB.

---

I suspect though that RGB calibration is not going to make much difference in an underwater environment. As the RGB image is much simpler than the depth image, the main consideration may be that there is enough light in an underwater environment for the RGB sensor to be able to see the under-the-surface environment.

A D455 type camera may be more suitable for the application, as it supports 'RGB from left infrared' mode, an RGB image that is generated by the same left-side infrared sensor that the depth image originates from. This means that the RGB image from the infrared sensor is perfectly aligned with the depth image and should also be able to take advantage of infrared light projected from the camera's IR emitter to illuminate the environment (the RGB sensor cannot see in dark environments).

D455 was used by the FishSense underwater project.

https://e4e.ucsd.edu/fishsense

https://kastner.ucsd.edu/wp-content/uploads/2021/08/admin/oceans21fishsense.pdf

[Zauzolkov]

@MartyG-RealSense Thank you for your answers.

---

In your kindly provided images of calibration, it looks as though the recommended target images are not being used for the appropriate calibration tool.

These images are only meant to demonstrate optical distortions and problem itself, not to show the entire calibration process.

---

Have you tried performing a calibration with the default unmodified graphical-interface Dynamic Calibration tool

Yes, we tried to perform Dynamic Calibration with "vertical bars" pattern (in water) using unmodified tool from DEB package. The results were not impressive. Custom Calibration showed much better results in underwater conditions.

Here are two recordings where you can see the effect after Custom Calibration, and you can also see the Checkerboard Pattern we used:

Show videos

rs-no-calib.mp4

rs-with-calib.mp4

Some constants of CalibrationManager.h have been modified to fit our pattern

const int NUM_SHOTS = 6;
const int CHESSBOARD_HEIGHT = 6;
const int CHESSBOARD_WIDTH = 9;
const float CHESSBOARD_SQUARE_SIZE = 30.0f;

---

The four-dot calibration image in your Examples of misalignment section is not used for calibrating depth quality

Yes, I understand. With this image I was just trying to demonstrate the alignment error between RGB and 3D mesh (and that there is no such misalignment when using IR as a texture source).

---

I suspect though that RGB calibration is not going to make much difference in an underwater environment. As the RGB image is much simpler than the depth image, the main consideration may be that there is enough light in an underwater environment for the RGB sensor to be able to see the under-the-surface environment.

There seems to be some misunderstanding. Please take a look at this video, which demonstrates the alignment problem. Calibration Pattern here is just an object for demonstration purposes only (It could have been any other object).

rs-misalign.mp4

Notice misalignment after changing Texture Source to RGB (instead of IR). Color texture is not "painted" correctly to 3D mesh. This misalignment persists even after rebooting the device, restarting realsense-viewer, etc.

---

Let me highlight the key points:

• I don't think there are any problems with the calculation/choosing of distortion coefficients. I am satisfied with the results of IR camera calibration, depth map is good after Custom Calibration.
• I see an alignment error when using RGB as a texture source.
• It seems (to me) that the source of this problem is that while IR cameras follow the distortion coefficients, but the RGB camera seems to "ignore" the DistortionRGB parameters.

We can even forget about some details like underwater use of the device; I am concerned about the following questions:

• Is there any way to set the DistortionRGB parameters so that the RGB camera will actually use these values ​​for un-distortion? (and not ignore these parameters)
• Is the misalignment issue related to the RGB camera's distortion settings being ignored? It seems that the RGB camera is not aligned with the IR cameras because of this issue. Am I wrong?
• Are there any recommendations for fixing the misalignment problem? (as shown in the last video)

---

On the topic of the D455 camera

A D455 type camera may be more suitable for the application, as it supports 'RGB from left infrared' mode, an RGB image that is generated by the same left-side infrared sensor that the depth image originates from.

Thanks, that sounds interesting. But I will note that after Factory Calibration Reset ("Gold" calibration settings) I do not see any problems with alignment. So it looks like D435 can handle the task.

also be able to take advantage of infrared light projected from the camera's IR emitter to illuminate the environment (the RGB sensor cannot see in dark environments).

This advantage is not relevant for us because of IR light absorption by water. We use an external laser projector to spread pattern (in the visible spectrum of light) and external light source for dark environments. Internal Projector of RealSense is always turned off.

Anyway, my current goal is to demonstrate the best results using the device we already have (hoping to get approval to use Intel photogrammetry solutions in the future).

[Zauzolkov]

I completed the full Dynamic Calibration sequence again, and here's what I got:

• The depth map quality is much worse than after Custom Calibration (curve/flat problem). The depth map is very similar to what I see with the Factory/Gold settings. Thus, the results of Dynamic Calibration are rather useless for me, since it doesn't undistort correcctly in underwater conditions.
• DistortionRGB parameters are empty (zero-filled) in both CalibrationTable.xml and RgbCalibrationTable.xml files:

<param name = "DistortionRGB">
    <value>0</value>
    <value>0</value>
    <value>0</value>
    <value>0</value>
    <value>0</value>
</param>

Although I did not skip the RGB camera calibration step…

I can provide video recordings to demonstrate the results if required (before Dynamic Calibration, after Dynamic Calibration and Custom Calibration).

Edit the Calibration file as follows…

I followed this advice. The depth map is now better than after the initial Dynamic Calibration, but slightly worse than Custom Calibration (so, I'll continue to use my own values calculated using Custom Calibration). I also noticed that the suggested values ​​are quite close to those calculated by Custom Calibration (so it looks like my Custom values ​​are really close to what I need).

Anyway, misalignment problem is still present even after following the mentioned advice.

Now I have a new idea

I noticed that "scale" of misalignment seems to depend on Focal Length, so the next idea came up: I will try to manually change the Focal Length parameter for IR cameras so that the depth map would better match the RGB texture. I'm not sure how correct this is and whether it will be successful, but I will try to do this.

Their advice covers custom depth calibration though, and they had not attempted doing so with RGB calibration values in as much detail, though they thought that the same principles as the depth calibration procedure would logically be applicable to RGB.

Unfortunately, this advice has nothing to do with my observation that the RGB camera still seems to ignore these parameters (DistortionRGB).

---

I still hope there are some ideas regarding the "misalignment" issue and/or the "RGB camera ignoring the settings" issue.

[Zauzolkov]

Here's another demonstration of the misalignment problem. This video was recorded in an aquarium at a short distance. In the pool — at a greater distance — things get dramatically worse (see video in this comment).

Here I am switching between IR and RGB texture sources to demonstrate misalignment.

rs-misalign-2.mp4

Here is another demonstration using rs-align example:

rs-misalign-via-example.mp4

There is no such problem after reset to Factory/Gold settings, but default settings aren't suitable for underwater conditions.

[MartyG-RealSense]

I found a case at #11693 where another RealSense user was having same issues with curvature and distortion with RGB. They had the D405 camera model, which does not have an RGB sensor and obtains its RGB image from the depth sensor.

Another RealSense user commented on that case at #11693 (comment) that their changes to RGB distortion values were not being updated.

By default on D435 cameras (the defaults are for in-the-air use, of course), all color distortion coefficients are set to zero, as advised at #1430 (comment)

I'm very appreciative of how patient you have been and the number of detailed examples and images / videos that you have provided. As Intel do not have the facilities to test cameras underwater though, it is unfortunately difficult to provide advice on this subject and the best source of advice is likely to be other users with experience of underwater calibration, like the ones at #11693

Regarding the problem of custom RGB non-zero values apparently not having an effect even in non-underwater situations, I will consult with my Intel RealSense colleagues.

[MartyG-RealSense]

I have received feedback from my RealSense colleagues regarding your case.

They concluded that the CustomCalibration was likely working as expected if the calibration table reflects the modifications that you have made, even if the RGB image does not appear to have been affected.

They advised that RealSense cameras do not undistort the RGB and the color image is sent distorted. This occurs even on RealSense camera models that have non-zero color coefficients such as D455 and D457.

My colleagues very recently created a Pull Request (PR) at #13833 for fixing the RGB distortion model used. It has not yet been incorporated into the RealSense SDK at the time of writing this though. The first SDK version that will likely have the fix once it is merged is the development branch of librealsense.

https://github.com/IntelRealSense/librealsense/tree/development

[Zauzolkov]

@MartyG-RealSense Thank you very much for your reply, I found it very helpful and informative. I especially appreciate that you asked for feedback on my issue from your colleagues.

My colleagues very recently created a Pull Request (PR) at #13833 for fixing the RGB distortion model used.

I applied the patch you mentioned (by modifying the d400-private.cpp file in RealSense SDK) and rebuilt RealSense-Viewer from sources. I also added a couple of additional log messages in the same file to make sure that the modified code is actually running.

Unfortunately, I do not see any changes in the RGB stream. Everything looks the same as in my first message: changes in distortion parameters affect only the IR camera streams, while RGB stream remains unchanged. I also tried reconnecting the device after writing the calibration settings, restarting the RealSense-Viewer and other similar actions.

I also noticed #13824, and it seems that this issue may have a similar root cause.

[Zauzolkov]

After further investigation of the problem, I found the following: after I modified the DistortionRGB parameters, even though the RGB camera stream remains unchanged in the video preview (the RGB Camera Color stream window in RealSense-Viewer), but the difference is visible in 3D mode, when the RGB texture is selected as texture source for the mesh. So it actually affects the alignment between the depth map and the IR/RGB textures, while there is no visible effect in video preview.

Thus, I arrive at the following conclusions:

• The DistortionLeft and DistortionRight parameters affect the IR camera’s video stream itself, and the undistortion is observable directly in the video preview (2D mode in RealSense-Viewer).

• The DistortionRGB parameters, however, don't appear to affect the video stream, so the image remains unchanged in video preview (2D mode).

• Nevertheless, the DistortionRGB parameters likely influence the alignment of RGB texture onto the 3D mesh. As a consequence, these parameters actually affect the convergence between the IR and RGB textures.

• I suppose that the impact of the DistortionRGB parameters can only be "seen" (visible) in 3D mode (or in those RealSense examples where streams alignment occurs).

• As you mentioned earlier in another issue:

  RealSense camera model's hardware applies a Brown-Conrady distortion model at the point of capture before the data is sent through the USB cable to the computer

  but apparently this statement only applies to IR cameras?

I would appreciate your feedback on whether my conclusions are correct or if I'm mistaken.

I'm also curious to know if the described behavior is expected for RealSense, or if it's a bug (that should have been fixed by the mentioned patch):

They advised that RealSense cameras do not undistort the RGB and the color image is sent distorted.

[MartyG-RealSense]

Thanks very much for the detailed information. I have fed your conclusions back to my colleagues to seek their input regarding whether they are correct.

The following distortion models are applied to the streams:

INFRARED SENSOR: Brown-Conrady (depth and infrared)

RGB SENSOR: Inverse Brown-Conrady (color)

Brown-Conrady distorts the image and Inverse Brown-Conrady undistorts the image.

[MartyG-RealSense]

My colleagues have confirmed that your conclusions are correct.

They add that "Whilst the Left and Right IR images are rectified by the (Vision Processor) D4, the RGB image is not (this is not a bug). Nevertheless, the RGB calibration params are used when performing 2D-3D projections and frame alignments".

https://github.com/IntelRealSense/librealsense/wiki/Projection-in-RealSense-SDK-2.0#d400-series

[Zauzolkov]

@MartyG-RealSense Thank you very much for your help! It seems that everything is much clearer now — it appears that the issue was due to the non-obvious behavior of RealSense. At the moment, I don't have any questions. If I have any new questions on this topic, I will reopen this Issue. Also, if I have any new observations or experiences regarding the subject, I will post them in this Issue without reopening it.

[MartyG-RealSense]

You are very welcome, @Zauzolkov - I'm pleased that I could help. Thanks very much for the update!