Multimodal Neural Radiance Fields for Thermal Novel View Synthesis
INFORMATION
Method
We propose a novel multimodal approach based on Neural Radiance Fields, capable of rendering new RGB and thermal views of a scene jointly.
Thermal images are textureless and suffer from the ghosting effect. To overcome the lack of texture in thermal images, we use paired RGB and thermal images to learn scene density, while distinct networks estimate color and temperature information.
Furthermore, we introduce ThermoScenes, a new dataset to palliate the lack of available RGB+thermal datasets for scene reconstruction.
Results
Experimental results validate that ThermoNeRF achieves accurate thermal image synthesis, with an average mean absolute error of 1.5C, an improvement of over 50% compared to using concatenated RGB+thermal data with Nerfacto, a state-of-the-art NeRF method.
What’s next
- ThermoNeRF assumes aligned RGB-thermal image pairs; handling misaligned or unpaired data would make capture far easier in the field.
- Speed and scale: extending from facades to whole buildings, and comparing against Gaussian-splatting-based thermal reconstruction.
- Both directions are addressed by follow-up work in the group: see SEAR (feed-forward transformers, unpaired capture) and Thermoxels (simulation-ready models).