用扩散模型实现大场景3D可控光照重渲染，效果逼真。
GR3EN: Generative Relighting for 3D Environments
- 将视频到视频的光照重渲染模型蒸馏到3D重建中，避开逆渲染难题。
- 在真实与合成数据集上均能生成新光照下的逼真视角图像。
- 适合需要高质量3D场景光照调整的研究者与开发者。
我们提出一种针对大规模室内场景3D重建的光照重渲染方法。现有3D场景光照重渲染方法常需解决欠定或病态的逆渲染问题，难以在复杂真实场景中生成高质量结果。尽管近期基于生成式图像和视频扩散模型的光照重渲染取得进展，但这些技术要么局限于2D图像与视频，要么仅适用于单个物体的3D光照重渲染。我们的方法通过将视频到视频光照重渲染扩散模型的输出蒸馏至3D重建，实现了对大规模场景的可控3D光照重渲染。该方法规避了复杂的逆渲染过程，构建了一个灵活系统，可对复杂真实场景的3D重建进行光照重渲染。我们在合成与真实世界数据集上验证了该方法，结果表明其能忠实呈现新光照条件下的新颖视图。
原文摘要 · Abstract (English)
We present a method for relighting 3D reconstructions of large room-scale environments. Existing solutions for 3D scene relighting often require solving under-determined or ill-conditioned inverse rendering problems, and are as such unable to produce high-quality results on complex real-world scenes. Though recent progress in using generative image and video diffusion models for relighting has been promising, these techniques are either limited to 2D image and video relighting or 3D relighting of individual objects. Our approach enables controllable 3D relighting of room-scale scenes by distilling the outputs of a video-to-video relighting diffusion model into a 3D reconstruction. This side-steps the need to solve a difficult inverse rendering problem, and results in a flexible system that can relight 3D reconstructions of complex real-world scenes. We validate our approach on both synthetic and real-world datasets to show that it can faithfully render novel views of scenes under new lighting conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。