用扩散模型统一光照,重建极端光照下的3D物体细节。
Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variation
- 先用多视角光照扩散模型统一输入图像的光照条件。
- 在统一光照下重建3D几何与外观,显著提升保真度。
- 特别擅长还原高光物体的视点依赖外观,适合真实场景建模。
从不同光照环境下拍摄的照片中重建物体的几何与外观极具挑战,尤其对镜面材质而言,其外观强烈依赖于观察方向。现有方法或使用每张图的嵌入向量建模外观变化,或采用基于物理的渲染恢复材质和光照,但在输入光照差异显著时,仍难以准确还原视点依赖的外观,结果多为泛化性较强的漫反射效果。本文提出一种新方法:首先利用多视角光照扩散模型将输入图像统一重光照至单一参考光照;随后采用对残余不一致性鲁棒的辐射场架构进行3D几何与外观重建。我们在合成与真实数据集上验证了该方法,结果表明其在极端光照变化条件下显著优于现有技术,尤其能有效恢复高光物体的视点依赖外观,而这是以往方法无法实现的。
原文摘要 · Abstract (English)
Reconstructing the geometry and appearance of objects from photographs taken in different environments is difficult as the illumination and therefore the object appearance vary across captured images. This is particularly challenging for more specular objects whose appearance strongly depends on the viewing direction. Some prior approaches model appearance variation across images using a per-image embedding vector, while others use physically-based rendering to recover the materials and per-image illumination. Such approaches fail at faithfully recovering view-dependent appearance given significant variation in input illumination and tend to produce mostly diffuse results. We present an approach that reconstructs objects from images taken under different illuminations by first relighting the images under a single reference illumination with a multiview relighting diffusion model and then reconstructing the object's geometry and appearance with a radiance field architecture that is robust to the small remaining inconsistencies among the relit images. We validate our proposed approach on both synthetic and real datasets and demonstrate that it greatly outperforms existing techniques at reconstructing high-fidelity appearance from images taken under extreme illumination variation. Moreover, our approach is particularly effective at recovering view-dependent "shiny" appearance which cannot be reconstructed by prior methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。