无需训练即可从稀疏图像重建3D场景并生成新视角，支持物体级编辑。
Training-Free Instance-Aware 3D Scene Reconstruction and Diffusion-Based View Synthesis from Sparse Images
- 通过图像扭曲检测异常点，实现鲁棒的点云重建。
- 用2D分割图引导3D实例提升，生成一致的实例感知3D表示。
- 结合扩散模型修复缺失几何，适合快速编辑与高质量渲染。
我们提出一种全新的无训练3D室内场景重建与视图合成系统，仅需少量无姿态约束的RGB图像即可完成。相比传统辐射场方法依赖密集视角和逐场景优化，本方法无需训练或姿态预处理，仍可实现高保真结果。系统包含三项关键创新：（1）基于图像扭曲的异常点剔除策略，增强点云重建的鲁棒性；（2）利用扭曲引导的2D到3D实例提升机制，将2D分割掩码映射为一致的实例感知3D表示；（3）将点云投影至新视角，并通过3D感知扩散模型精炼渲染结果。该方法借助扩散模型弥补几何缺失，显著提升稀疏输入下的真实感。此外，仅修改点云即可自然实现物体移除等实例级编辑，生成一致且无需重训练的新视图。实验表明，该方法为无需场景定制优化的高效可编辑3D内容生成开辟了新路径。
原文摘要 · Abstract (English)
We introduce a novel, training-free system for reconstructing, understanding, and rendering 3D indoor scenes from a sparse set of unposed RGB images. Unlike traditional radiance field approaches that require dense views and per-scene optimization, our pipeline achieves high-fidelity results without any training or pose preprocessing. The system integrates three key innovations: (1) A robust point cloud reconstruction module that filters unreliable geometry using a warping-based anomaly removal strategy; (2) A warping-guided 2D-to-3D instance lifting mechanism that propagates 2D segmentation masks into a consistent, instance-aware 3D representation; and (3) A novel rendering approach that projects the point cloud into new views and refines the renderings with a 3D-aware diffusion model. Our method leverages the generative power of diffusion to compensate for missing geometry and enhances realism, especially under sparse input conditions. We further demonstrate that object-level scene editing such as instance removal can be naturally supported in our pipeline by modifying only the point cloud, enabling the synthesis of consistent, edited views without retraining. Our results establish a new direction for efficient, editable 3D content generation without relying on scene-specific optimization. Project page: https://jiatongxia.github.io/TID3R/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。