Integrating TAL with Mitsuba 3 for NLOS Data Rendering

[Lingyun-Qiu]

Hi,

I am currently working on a project that involves rendering NLOS (Non-Line-of-Sight) data using TAL and Mitsuba 3. I have compiled Mitsuba 3 following the instructions provided at Mitsuba documentation. Here are the steps I took and the issues I encountered:

System Information:

• Operating System: Ubuntu 22.04.4 LTS
• Clang Version: Ubuntu clang version 15.0.7
• Python Version: 3.12.3
• Ninja Version: 1.10.1
• CMake Version: 3.22.1
• LLVM Config Version: 14.0.0

1. Compilation of Mitsuba 3:

   • I chose to compile Mitsuba 3 rather than using the pip installation because I wanted to enable specific variants such as llvm_ad_mono and cuda_mono.

2. Setting Environment Variables:

   • After compilation, I successfully configured the environment variables using source setpath.sh. At this point, I was able to import mitsuba but encountered an issue where I couldn't import mitransient, with an error indicating that the module was not found.

3. Installing mitransient:

   • I downloaded and installed the latest version of mitransient (version 1.0.3) from GitHub. To ensure compatibility, I uninstalled the included mitsuba3 package and verified that my custom-compiled mitsuba3 was being used.

4. Installing TAL:

   • I downloaded and installed the latest version of TAL from Diego Royo's GitHub repository.

5. Encountered Issues:

   • Despite the installations, I faced several issues:
     • mitsuba.__version__ was not defined.
     • ModuleNotFoundError: No module named 'mitsuba.math' occurred when calling render_steady('back_view', 0) in /tal/tal/render/render.py at line 266.

I am seeking guidance on how to properly integrate TAL with my custom-compiled Mitsuba 3 to achieve NLOS data rendering. Any help or pointers would be greatly appreciated!

Thank you in advance for your assistance.

Best regards,

Lingyun Qiu

[diegoroyo]

Hi

Thanks for your detailed report. Can you post the output of tal -v? It should tell you whether it's locating the mitsuba and mitransient packages, and if so, where. Here's the output that I get as an example:

❯ tal -v
tal v0.16.2 (/home/droyo/.pyenv/versions/3.11.3/envs/mitsuba3-py3.11/lib/python3.11/site-packages/tal)

* Render backend: mitsuba3-transient-nlos
mitransient v1.0.3 (/home/droyo/.pyenv/versions/3.11.3/envs/mitsuba3-py3.11/lib/python3.11/site-packages/mitransient)
mitsuba v3.5.0 (/home/droyo/.pyenv/versions/3.11.3/envs/mitsuba3-py3.11/lib/python3.11/site-packages/mitsuba)

Note that tal -v is not available in the PIP version of tal yet, only in the latest commits of the GitHub repo.

Finally, if you keep having issues, could you also provide a yaml file of an example scene that gives you the same errors?

Best
Diego

[Lingyun-Qiu]

Hi Diego,

Thanks for your response. Here's the output of tal -v from my environment:

tal -v
tal v0.16.3 (/home1/newhome/lingyun/libs/tal/tal)

* Render backend: mitsuba3-transient-nlos
mitransient v1.0.3 (/home1/newhome/lingyun/libs/mitsuba3-transient-nlos-main/mitransient)
mitsuba v3.6.0 (/home1/newhome/lingyun/libs/mitsuba3/build/python/mitsuba)

Please note that the mitsuba v3.6.0 is manually set by modifying /home1/newhome/lingyun/libs/mitsuba3/build/python/mitsuba/__init__.py. As mentioned earlier, I encountered an error related to the missing __version__ attribute after compiling Mitsuba3. I resolved this by adding the following lines to the __init__.py file:

# Manually set the version attribute
mitsuba_alias.__version__ = '3.6.0'

For testing, I used the YAML file provided in the TAL GitHub repository located at confocal-scene.yaml. I encountered the following error message during rendering:

tal render confocal-scene/confocal-scene.yaml
Found a render in progress (20240609-110921), continuing...
back_view for confocal-scene steady render...
/home1/newhome/lingyun/tests/tal_test/render-reconstruct-confocal/confocal-scene/20240609-110921/partial/confocal-scene_back_view.npy
Process Process-1:
Traceback (most recent call last):
  File "/home1/newhome/lingyun/.conda/envs/mitsuba3/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home1/newhome/lingyun/.conda/envs/mitsuba3/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home1/newhome/lingyun/libs/tal/tal/render/mitsuba3_transient_nlos.py", line 592, in run_mitsuba
    sys.stdout = sys.__stdout__
                 ^^^
UnboundLocalError: cannot access local variable 'sys' where it is not associated with a value
Traceback (most recent call last):
  File "/home1/newhome/lingyun/.local/bin/tal", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home1/newhome/lingyun/libs/tal/tal/__main__.py", line 174, in main
    render_nlos_scene(config_file, args)
  File "/home1/newhome/lingyun/libs/tal/tal/render/__init__.py", line 30, in render_nlos_scene
    render.render_nlos_scene(config_path, args)
  File "/home1/newhome/lingyun/libs/tal/tal/render/render.py", line 266, in render_nlos_scene
    render_steady('back_view', 0)
  File "/home1/newhome/lingyun/libs/tal/tal/render/render.py", line 258, in render_steady
    raise e
ModuleNotFoundError: No module named 'mitsuba.math'

Best,
Lingyun Qiu

[diegoroyo]

Hi again,

Sorry for the delayed response, I have been away. I have tried to reproduce your issue:

• Python 3.12.3
• mitsuba 3.5.2
• mitransient 1.0.3
• y-tal 0.17.0 (from GitHub last commit)
• ninja 1.11.1
• cmake 3.28.1

but it works correctly on my end and cannot repoduce your error. Nevertheless I have multiple suggestions below, you can follow my installation procedure and see if it helps you. If you have additional questions or want me to try something do not hesitate to ask.

I got a clean Python 3.12.3 installation (test-mi352-py3.12, as you will see in the code below) and:

1. Compiled mitsuba 3.5.2 using:

cmake -GNinja -DCMAKE_CXX_COMPILER=/usr/bin/g++-12 -DCMAKE_C_COMPILER=/usr/bin/gcc-12 -DCMAKE_BUILD_TYPE=Release -DPython_EXECUTABLE=/home/droyo/.pyenv/versions/test-mi352-py3.12/bin/python ..

on the mitsuba/build folder. Check CMake output to verify that it detected your Python version and Dr.JIT is building the Python plugin.

2. At this point, you can source mitsuba/build/setpath.sh and you will be able to import mitsuba and get its version. Make sure that you can do this before installing mitransient or tal.

>>>import mitsuba as mi
>>> mi.set_variant('llvm_mono')
>>> print(mi.__version__)
3.5.2

3. With this, I can pip install mitransient --no-dependencies. The --no-dependencies part is important if you want to use your own mitsuba installation, to prevent the installation of the mitsuba python package. You can also install tal (last version on GitHub) by cloning and then pip install wheel && scripts/local_install.sh. Or you can pip install y-tal (which gives you v0.16.1). Both should work

Using this setup, tal render confocal-scene works fine.

In summary, I cannot reproduce your issue but I believe that it may be caused by an incorrect mitsuba installation (at least the missing version thing, which seems weird)

Best
Diego

[Kaixin-ww]

Hi Diego!
Recently, I used mitransient based on Mitsuba3 to obtain transient images from z.obj：

However, the references provided by mitransient use non-confocal scanning. Now I want to use the tal tool to achieve confocal data acquisition. Could you please guide me on how to do this? I am using PyCharm Community Edition on Windows, and have already installed tal in my Python environment.

(rendering) C:\Users\81248>tal -v
tal v0.19.0 (D:\anaconda\envs\rendering\lib\site-packages\tal)
Render backend: mitransient (mitsuba3-transient-nlos)
mitransient v1.1.2 (D:\anaconda\envs\rendering\lib\site-packages\mitransient)
mitsuba v3.6.4 (D:\anaconda\envs\rendering\lib\site-packages\mitsuba)

Looking forward to your reply.
Best，
Kaixin

[diegoroyo]

Hi Kaixin, thanks for checking out tal+mitransient :)

To simulate confocal data acquisition, you'll need to edit the .yaml file that was generated. Go to the section labeled:

##
# Laser and sensor
##
scan_type: single  # single, confocal or exhaustive
...
sensor_width: 256
sensor_height: 256
...
laser_width: 32
laser_height: 32

You need to change scan_type to confocal. If the lines are commented (#scan_type: ...), uncomment them by removing the #.

By default, scan_type: single uses only one laser position and a 256x256 grid of sensor positions (sensor_width x sensor_height).

For scan_type: confocal if you want to scan a 64x64 grid of points you should edit sensor_width: 64 and laser_width: 64, and sensor_height: 64 and laser_height: 64. It is important that the sensor and laser width and height match.

Finally you can run the same tal render <Z> command as before, it will read the new yaml file and generate a confocal result. It's normal if it takes a bit longer than the non-confocal result.

Let me know if you need help with anything else!

[Kaixin-ww]

Thank you so much, Diego!
Your responses are always incredibly timely. It's an honor to have such an outstanding developer like you in the relevant research field. ：）Thanks to your guidance, I've successfully obtained transient images via the command line. I currently have a few more questions and hope you can guide me:
1：I've found TAL extremely convenient. Does this mean that in most scenarios, I no longer need to independently call the Mitsuba and mitransient modules?
2：When I set num_bins: 1024, bin_width_opl: 0.0012, and start_opl: 1.95, comparing transient results between confocal and non-confocal configurations, I noticed that transient images from the confocal setup resemble the original scene closely in certain frames, while the non-confocal ones do not. Is this confocal result correct? What causes this discrepancy?

3：Can I implement the rendering process via a PyCharm script? It seems difficult to control the addresses of input and output files through the command line.
Looking forward to your reply.
Best regards,

[diegoroyo]

Thanks a lot! It means a lot. I'll answer in the same order:

1. Yes, for most cases (e.g. to simulate a few NLOS setups) it's easier to do through tal in the command line. You can achieve the same by calling Mitsuba/mitransient directly, but it's probably harder (see my answer # 3)

2. Your result is correct. Think about the paths that photons take from each illuminated point to each captured point in the relay wall. In a confocal setting, the first photons that arrive to the camera (i.e. the first non-zero frames of the video) are photons that travel in a direction perpendicular to the relay wall and back, as these are the shortest paths. Because the Z is positioned coplanar to the relay wall this has the effect that in the first frames you clearly see the Z in the video. In non-confocal setups this does not happen as the photons' time of flight do not coincide in this way.

3. You could do two things:
   a) In your Python script, execute the tal render <scene> command and then copy/move the input/output files from/to another folder. This is probably the easiest.
   b) Copy the nlos_scene.xml file that tal render <scene> generates, and pass it to scene = mi.load_file('nlos_scene.xml'). Then by calling data_steady, data_transient = mi.render(scene, spp=<spp>) you obtain the data_transient which is the histogram/H file that you would obtain with tal. This example script can help you if you do this.

[Kaixin-ww]

Thank for your reply！

Your explanations have clarified a lot of my confusion, especially regarding the confocal setup and the appearance of the Z in the first frames—it makes perfect sense now.

I appreciate the two practical solutions you provided for integrating the rendering process into my workflow; option a) sounds straightforward, and I’ll definitely give it a try.

Thanks again for taking the time to help me out—it means a lot !I will recommend your work to the teachers and students around me.：）