用扩散模型从单张图重建完整3D场景，无需多视角数据
Enhancing Monocular 3D Scene Completion with Diffusion Model
- 用视觉语言模型生成场景描述，引导扩散模型生成多视角图像
- 在ScanNet等数据集上实现高质量3D重建，优于现有单目方法
- 无需额外训练，适合虚拟现实与自动驾驶场景建模
3D场景重建对虚拟现实、机器人和自动驾驶至关重要，使机器能理解复杂环境。传统3D高斯泼溅技术依赖多视角图像以达最优性能，但限制了单图场景的应用。本文提出FlashDreamer，一种仅需单张图像即可重建完整3D场景的新方法，显著减少对多视角输入的依赖。该方法利用预训练视觉-语言模型生成场景描述提示，引导扩散模型生成多个视角图像，并融合形成一致的3D重建。大量实验表明，该方法能有效且稳健地将单图输入扩展为完整3D场景，无需额外训练，大幅拓展了单目3D重建能力。代码已开源：https://github.com/CharlieSong1999/FlashDreamer/tree/main。
原文摘要 · Abstract (English)
3D scene reconstruction is essential for applications in virtual reality, robotics, and autonomous driving, enabling machines to understand and interact with complex environments. Traditional 3D Gaussian Splatting techniques rely on images captured from multiple viewpoints to achieve optimal performance, but this dependence limits their use in scenarios where only a single image is available. In this work, we introduce FlashDreamer, a novel approach for reconstructing a complete 3D scene from a single image, significantly reducing the need for multi-view inputs. Our approach leverages a pre-trained vision-language model to generate descriptive prompts for the scene, guiding a diffusion model to produce images from various perspectives, which are then fused to form a cohesive 3D reconstruction. Extensive experiments show that our method effectively and robustly expands single-image inputs into a comprehensive 3D scene, extending monocular 3D reconstruction capabilities without further training. Our code is available https://github.com/CharlieSong1999/FlashDreamer/tree/main.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。