Confocal non-line-of-sight simulation bug based on mitransient

[Kaixin-ww]

Dear Diego,

Since it is not very convenient to perform confocal reconstruction using tal, I have recently been trying to use a loop method to conduct confocal simulations with mitransient. In accordance with the instructions provided here, I have set the corresponding parameters.

I saved the rendered TOF data as .mat files to facilitate reconstruction using the LCT method in MATLAB. The reconstruction was successful for the letter "Z" example. I then performed the same rendering and reconstruction steps on the letter "H" that I created, but the correct results do not seem to be achievable. What are your thoughts on this outcome?



I'm looking forward to your reply

[diegoroyo]

If you want to generate confocal datasets with your own code that directly calls mitransient, there's an important implementation detail about simulations.

Consider a laser emitter placed at position L that is pointed towards a point X on the relay wall. In a confocal setup L remains the same and X scans the whole wall. The way light transport simulation is implemented, light emitted from the laser suffers a (cos theta)*(1/d)^2 falloff term, where d = distance(L, X) and theta is the angle between (L - X) and the relay wall's normal. So every simulation is compensated with a (1/cos theta)*d^2 term. If you plan to use your own code you should do that too. This is implemented in tal.reconstruct.compensate_laser_cos_dsqr(data: NLOSCaptureData). In practice, you should do sth like this

import tal

data = tal.io.read_capture(...)
data = tal.reconstruct.compensate_laser_cos_dsqr(data)

tal.reconstruct.fbp.solve(...)

The default position for the laser position L is a bit different from real NLOS setups so this effect is more noticeable than in real datasets.

So, in summary:
Try doing that compensation and let me know if it works.
Also, out of curiosity, why are you implementing your own loop instead of using tal? What should be changed in the tal repository to match what you need? I'm saying this because tal actually implements a loop too.

[Kaixin-ww]

Let me first address your curiosity. The first reason I use my own loop instead of tal is that I haven't used the .hdf5 files generated by tal before. Apart from directly calling them with tal, I don't know how to more intuitively view the rendered TOF data. However, the result of calling mitransient through my own loop is a Tensor, and I can even import it into MATLAB to verify it using some previous reconstruction algorithms. This is because many classic NLOS works are implemented in MATLAB.

The second reason is that I want to reproduce the work of a paper[]https://doi.org/10.1145/3680528.3687575, involves the use of polarization in mitsuba. Since tal is highly integrated, I'm not sure how to perform polarized NLOS rendering through tal. To facilitate potential usages in the future, I want to first try using my own loop. Maybe tal can also incorporate polarization; it's just that I haven't reached the level of understanding to do so yet.

I also tried the method you mentioned, but the input for data = tal.io.read_capture(...) has to be an .hdf5 file. As I said earlier, I don't know how to work with such files. For now, I only know how to use the .hdf5 files automatically generated by tal. So I rendered my letter H scene again using tal and performed the reconstruction with the method you suggested, and this is the result I got.

It seems to be somewhat similar to yesterday's result, haha—part of the shape is there, but it's not complete. However, when using the built-in letter Z example, this method indeed produces a good reconstruction effect with lower noise.

To summarize my current question: If I want to implement the polarized NLOS as described in the paper, can I do it directly using tal? It's really powerful.

[Kaixin-ww]

Hi Diego,
Thank you for your guidance! Last week, I tried the compensation method you mentioned, and it really worked. Besides, I also studied the .hdf5 file generated by Tal. I exported the compensated TOF data to MATLAB and performed reconstruction using the LCT method, and the result was correct. I found that the reason why the result for the letter "H" was incorrect last time was that the normals of the obj model itself were not adjusted. Now, correct results can be obtained.

In addition, I tried to carry out confocal scanning with my own loop and added compensation during the loop. The result is better than before, but not as good as that of the data generated by Tal, with some artifacts appearing.

Here is my Python file and would like to ask you to guide me on any obvious errors in it. Many thanks！

import mitsuba as mi
mi.set_variant('cuda_ad_mono')
import scipy.io as sio
import mitransient as mitr
import numpy as np

wall_scale=1
laser_xyz=np.array([-0.25, 0.0, 0.25])
geometry = mi.load_dict(
    {
        "type": "obj",
        "filename": "./Z.obj",
        "to_world": mi.ScalarTransform4f().translate([0, 0.0, 1]),
        "bsdf": {"type": "diffuse", "reflectance": 1.0},
    }
)
# Load the emitter (laser) of the scene
emitter = mi.load_dict(
    {
        "type": "projector",
        "irradiance": 1.0,
        "fov": 0.1,
        "to_world": mi.ScalarTransform4f().translate(laser_xyz),
    }
)
# Define the transient film which store all the data
transient_film = mi.load_dict(
    {
        "type": "transient_hdr_film",
        "width": 1,
        "height": 1,
        "temporal_bins": 1024,
        "bin_width_opl": 0.0048,
        "start_opl": 0.2,
        "rfilter": {"type": "box"},
    }
)
# Define the sensor of the scene
nlos_sensor = mi.load_dict(
    {
        "type": "nlos_capture_meter",
        "sampler": {"type": "independent", "sample_count": 25_000},
        "account_first_and_last_bounces": False,
        "sensor_origin": mi.ScalarPoint3f(-0.25, 0.0, 0.25),
        "transient_film": transient_film,
        "original_film_width": 64,
        "original_film_height": 64,
    }
)
# Load the relay wall. This includes the custom "nlos_capture_meter" sensor which allows to setup measure points directly on the shape and importance sample paths going through the relay wall.
relay_wall = mi.load_dict(
    {
        "type": "rectangle",
        "bsdf": {"type": "diffuse", "reflectance": 1.0},
        "nlos_sensor": nlos_sensor,
        "to_world": mi.ScalarTransform4f().scale([wall_scale,wall_scale,wall_scale]),

    }
)
# Finally load the integrator
integrator = mi.load_dict(
    {
        "type": "transient_nlos_path",
        "filter_bounces": 1,
        "discard_direct_paths": False,
        "nlos_laser_sampling": True,
        "nlos_hidden_geometry_sampling": True,
        "nlos_hidden_geometry_sampling_do_rroulette": False,
        "nlos_hidden_geometry_sampling_includes_relay_wall": True,
        "temporal_filter": "box",
    }
)
# Assemble the final scene
scene = mi.load_dict({
    'type': 'scene',
    'geometry' : geometry,
    'emitter' : emitter,
    'relay_wall' : relay_wall,
    'integrator' : integrator
})

width = 64
height = 64
results = np.zeros((width, height, 1024))
x = np.linspace(-1, 1, width)
y = np.linspace(-1, 1, height)
x_grid, y_grid = np.meshgrid(x, y)
z_grid = np.zeros_like(x_grid)
laser_grid_xyz = np.stack([x_grid, y_grid, z_grid], axis=-1)
grid_normal = np.array([0, 0, 1])

for i in range(width):
    for j in range(width):
        pixel=mi.Point2f(i,j)
        mitr.nlos.focus_emitter_at_relay_wall_pixel(pixel, relay_wall, emitter)
        _, data = mi.render(scene)

        results[i, j, :] = np.array(data)[0, 0, :, 0]
        w_i=laser_grid_xyz[i,j,:]-laser_xyz
        d = np.linalg.norm(w_i)
        w_i = w_i / d
        cos_term = np.dot(w_i, grid_normal)
        results[i, j, :] /= cos_term / d ** 2
        print(f"({i},{j}) scan complete")