Abstract



We present MIRReS, a novel two-stage inverse rendering framework that jointly reconstructs and optimizes explicit geometry, materials, and lighting from multi-view images. Unlike previous methods that rely on implicit irradiance fields or oversimplified path tracing algorithms, our method first extracts an explicit triangular mesh in the initial stage. Subsequently, it employs a more realistic physically-based inverse rendering model in the second stage, utilizing multi-bounce path tracing and Monte Carlo integration. By leveraging multi-bounce path tracing, our method not only effectively estimates indirect illumination--including self-shadowing and internal reflections--but also enhances the intrinsic decomposition of shape, material, and lighting. Moreover, we incorporate reservoir sampling into our framework to address the noise in Monte Carlo integration, enhancing convergence and facilitating gradient-based optimization with low sample counts. Through both qualitative and quantitative assessments across various scenarios, especially those with complex shadows, we demonstrate that our method achieves state-of-the-art performance in decomposition results. Additionally, our optimized explicit geometry supports further applications in scene editing, relighting, and material editing, compatible with modern graphics engines and CAD software.

Quantitative comparison

Image Note that in terms of novel view synthesis, our method demonstrates similar performance compared to TensoIR and GS-IR. This is because these baselines tend to overfit the training dataset and bake the complex lighting effects (eg. shadows) within the materials, as qualitatively analyzed above. Despite this, we can produce considerably better material estimation and relighting results.

Albedo results and relighting results

GT OURS TensoIR
GT OURS TensoIR
GT OURS NVDiffRecMC
Drag the slider to compare!

Albedo results

Image

Normal results

Image

visibility results

Image

Pipeline


This video will explain our motivation and each stage of the pipeline, and showcase our results.


Overview

Image



Results on TensoIR-Synthetic Dataset


Relighting results





Benefits


Scene editing and material editing



ReSTIR result under 1 spp and indirect shading effects

Image


BibTeX(arXiv)


            
              @misc{dai2024mirres,
                title={MIRReS: Multi-bounce Inverse Rendering using Reservoir Sampling}, 
                author={Yuxin Dai and Qi Wang and Jingsen Zhu and Dianbing Xi and Yuchi Huo and Chen Qian and Ying He},
                year={2024},
                eprint={2406.16360},
                archivePrefix={arXiv},
                primaryClass={id='cs.CV' full_name='Computer Vision and Pattern Recognition' is_active=True alt_name=None in_archive='cs' is_general=False description='Covers image processing, computer vision, pattern recognition, and scene understanding. Roughly includes material in ACM Subject Classes I.2.10, I.4, and I.5.'}
          }
        

Acknowledgements

Some code for this website was borrowed from TensoIR and FuseSR.