Abstract



We present MIRReS, a novel two-stage inverse rendering framework that jointly reconstructs and optimizes explicit geometry, materials, and lighting from multi-view images. Unlike previous methods that rely on implicit irradiance fields or oversimplified path tracing algorithms, our method first extracts an explicit triangular mesh in the initial stage. Subsequently, it employs a more realistic physically-based inverse rendering model in the second stage, utilizing multi-bounce path tracing and Monte Carlo integration. By leveraging multi-bounce path tracing, our method not only effectively estimates indirect illumination--including self-shadowing and internal reflections--but also enhances the intrinsic decomposition of shape, material, and lighting. Moreover, we incorporate reservoir sampling into our framework to address the noise in Monte Carlo integration, enhancing convergence and facilitating gradient-based optimization with low sample counts. Through both qualitative and quantitative assessments across various scenarios, especially those with complex shadows, we demonstrate that our method achieves state-of-the-art performance in decomposition results. Additionally, our optimized explicit geometry supports further applications in scene editing, relighting, and material editing, compatible with modern graphics engines and CAD software.

Albedo results and relighting results

GT OURS TensoIR
GT OURS TensoIR
GT OURS NVDiffRecMC

Normal results

Image

Pipeline


This video will explain our motivation and each stage of the pipeline, and showcase our results.


Overview

Image



Results on TensoIR-Synthetic Dataset


Relighting results



Albedo comparison

Image


Normal comparison

Image


Benefits


Scene editing and material editing



ReSTIR result under 1 spp and indirect shading effects

Image


BibTeX(arXiv)


            
              @misc{dai2024mirres,
                title={MIRReS: Multi-bounce Inverse Rendering using Reservoir Sampling}, 
                author={Yuxin Dai and Qi Wang and Jingsen Zhu and Dianbing Xi and Yuchi Huo and Chen Qian and Ying He},
                year={2024},
                eprint={2406.16360},
                archivePrefix={arXiv},
                primaryClass={id='cs.CV' full_name='Computer Vision and Pattern Recognition' is_active=True alt_name=None in_archive='cs' is_general=False description='Covers image processing, computer vision, pattern recognition, and scene understanding. Roughly includes material in ACM Subject Classes I.2.10, I.4, and I.5.'}
          }
        

Acknowledgements

Some code for this website was borrowed from TensoIR and FuseSR.