Fireflies is a wrapper for the Mitsuba Renderer and allows for rapid prototyping and generation of physically-based renderings and simulation data in a differentiable manner. It can be used for example, to easily generate highly realistic medical imaging data for medical machine learning tasks or (its intended use) test the reconstruction capabilities of Structured Light projection systems in simulated environments. I originally created it to research if the task of finding an optimal point-based laser pattern for structured light laryngoscopy can be reformulated as a gradient-based optimization problem.
This code accompanies the paper Fireflies: Photorealistic Simulation and Optimization of Structured Light Endoscopy accepted at SASHIMI 2024. 🎊
- Easy torch-like and pythonic scene randomization description. This library is made to be easily usable for everyone who regularly uses python and pytorch. We implement train() and eval() functionality from the get go.
- Integratable into online deep-learning and machine learning tasks due to the differentiability of the mitsuba renderer w.r.t. the scene parameters.
- Simple animation description. Have a look into the examples.
- Single Shot Structured Light specific. You can easily test different projection pattern and reconstruction algorithms on randomized scenes, giving a good estimation of the quality and viability of patterns/systems/algorithms.
Make sure to create a conda environment first. I tested fireflies on Python 3.10, it should however work with every Python version that is also supported by Mitsuba and Pytorch. I'm working on adding Fireflies to PyPi in the future. First install the necessary dependencies:
pip install pywavefront geomdl
pip install torch
pip install mitsuba
To run the examples, you also need OpenCV:
pip install opencv-python
Finally, you can install Fireflies via:
git clone https://github.com/Henningson/Fireflies.git
cd Fireflies
pip install .
import mitsuba as mi
import fireflies as ff
mi_scene = mi.scene(path)
mi_params = mi.traverse(mi_scene)
ff_scene = ff.scene(mi_params)
mesh = ff_scene.mesh_at(0)
mesh.rotate_z(-3.141, 3.141)
ff_scene.eval()
#ff_scene.train() generates uniformly sampled results on the right
for i in range(0, 20):
ff_scene.randomize()
mi.render(mi_scene)
A bunch of different examples can be found in the examples folder. Papercode can be found in the Paper branch.
You can easily generate a scene using Blender.
To export a scene in Mitsubas required .xml format, you first need to install the Mitsuba Blender Add-On.
You can then export it under the File -> Export Tab.
Make sure to tick the ✅ export ids Checkbox, as fireflies infers the object type by checking for name qualifiers with specific keys, e.g.: "mesh", "brdf", etc.
Can be found in the README of the paper branch.
Because optimizing a point-based laser pattern looks like fireflies that jet around. :)
A big thank you to Wenzel Jakob and team for their wonderful work on the Mitsuba renderer. You should definitely check out their work: Mitsuba Homepage, Mitsuba Github.
Furthermore, this work was supported by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant STA662/6-1, Project-ID 448240908 and (partly) funded by the DFG – SFB 1483 – Project-ID 442419336, EmpkinS.
Please cite this, if this work helps you with your research:
@InProceedings{10.1007/978-3-031-73281-2_10,
author="Henningson, Jann-Ole and Veltrup, Reinhard and Semmler, Marion and D{\"o}llinger, Michael and Stamminger, Marc",
title="Fireflies: Photorealistic Simulation and Optimization of Structured Light Endoscopy",
booktitle="Simulation and Synthesis in Medical Imaging",
year="2025",
publisher="Springer Nature Switzerland",
address="Cham",
pages="102--112",
isbn="978-3-031-73281-2"
}