- Yuxin Dai
- Qi Wang
- Jingsen Zhu
- Dianbing Xi
- Yuchi Huo
- Chen Qian
- Ying He
Abstract
We present MIRReS, a novel two-stage inverse rendering framework that jointly reconstructs and optimizes explicit geometry, materials, and lighting from multi-view images. Unlike previous methods that rely on implicit irradiance fields or oversimplified path tracing algorithms, our method first extracts an explicit triangular mesh in the initial stage. Subsequently, it employs a more realistic physically-based inverse rendering model in the second stage, utilizing multi-bounce path tracing and Monte Carlo integration. By leveraging multi-bounce path tracing, our method not only effectively estimates indirect illumination--including self-shadowing and internal reflections--but also enhances the intrinsic decomposition of shape, material, and lighting. Moreover, we incorporate reservoir sampling into our framework to address the noise in Monte Carlo integration, enhancing convergence and facilitating gradient-based optimization with low sample counts. Through both qualitative and quantitative assessments across various scenarios, especially those with complex shadows, we demonstrate that our method achieves state-of-the-art performance in decomposition results. Additionally, our optimized explicit geometry supports further applications in scene editing, relighting, and material editing, compatible with modern graphics engines and CAD software.
Quantitative comparison
Note that in terms of novel view synthesis, our method demonstrates similar performance compared to TensoIR and GS-IR. This is because these baselines tend to overfit the training dataset and bake the complex lighting effects (eg. shadows) within the materials, as qualitatively analyzed above. Despite this, we can produce considerably better material estimation and relighting results.
Albedo results and relighting results
Drag the slider to compare!
Albedo results
Normal results
visibility results
Pipeline
This video will explain our motivation and each stage of the pipeline, and showcase our results.
Overview
Results on TensoIR-Synthetic Dataset
Relighting results
Benefits
Scene editing and material editing
ReSTIR result under 1 spp and indirect shading effects
BibTeX(arXiv)
@misc{dai2024mirres,
title={MIRReS: Multi-bounce Inverse Rendering using Reservoir Sampling},
author={Yuxin Dai and Qi Wang and Jingsen Zhu and Dianbing Xi and Yuchi Huo and Chen Qian and Ying He},
year={2024},
eprint={2406.16360},
archivePrefix={arXiv},
primaryClass={id='cs.CV' full_name='Computer Vision and Pattern Recognition' is_active=True alt_name=None in_archive='cs' is_general=False description='Covers image processing, computer vision, pattern recognition, and scene understanding. Roughly includes material in ACM Subject Classes I.2.10, I.4, and I.5.'}
}