用扩散模型自动匹配光影，让3D点云物体自然融入场景。
Diffusion Models are Secretly Zero-Shot 3DGS Harmonizers
- 利用扩散模型隐式理解真实光照，指导物体光影修正。
- 通过新设计的得分优化目标，实现2.0 dB PSNR提升的重光照效果。
- 适合需要高质量3D物体插入的渲染与重建任务。
高斯溅射已成为多种3D计算机视觉任务的流行技术，包括新视角合成、场景重建和动态场景渲染。然而，如何让插入物体的外观与场景自然融合仍是一个未解难题。本文提出一种名为D3DR的方法，可在保持3DGS参数化结构的前提下，将一个3DGS物体插入到现有3DGS场景中，并修正其光照、阴影等视觉失真。我们发现，在大规模真实数据上训练的扩散模型具备隐式理解正确场景光照的能力，并将其引入我们的流水线。在物体插入后，通过优化受扩散基差分去噪分数（DDS）启发的目标函数，调整3D高斯参数以实现光照校正。我们提出一种新颖的扩散个性化技术，可保留物体在不同光照条件下的几何与纹理一致性，确保原始与插入物体的身份匹配。最终实验表明，该方法相比现有方法在重光照质量上提升2.0 dB PSNR。
原文摘要 · Abstract (English)
Gaussian Splatting has become a popular technique for various 3D Computer Vision tasks, including novel view synthesis, scene reconstruction, and dynamic scene rendering. However, the challenge of natural-looking object insertion, where the object's appearance seamlessly matches the scene, remains unsolved. In this work, we propose a method, dubbed D3DR, for inserting a 3DGS-parametrized object into a 3DGS scene while correcting its lighting, shadows, and other visual artifacts to ensure consistency. We reveal a hidden ability of diffusion models trained on large real-world datasets to implicitly understand correct scene lighting, and leverage it in our pipeline. After inserting the object, we optimize a diffusion-based Delta Denoising Score (DDS)-inspired objective to adjust its 3D Gaussian parameters for proper lighting correction. We introduce a novel diffusion personalization technique that preserves object geometry and texture across diverse lighting conditions, and utilize it to achieve consistent identity matching between original and inserted objects. Finally, we demonstrate the effectiveness of the method by comparing it to existing approaches, achieving 2.0 dB PSNR improvements in relighting quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。