用视觉定位增强场景回归，实现大规模三维重建
SAIL-Recon: Large SfM by Augmenting Scene Regression with Localization
- 通过锚点图像构建神经场景表征，引导全图重建
- 在TUM-RGBD等数据集上达到最优姿态估计与新视角合成效果
- 适合处理海量图像的大规模三维重建任务
场景回归方法（如VGGT）通过直接从输入图像回归相机位姿和三维场景结构来解决结构光运动（SfM）问题，在极端视角变化下表现优异。然而，这些方法难以处理大量输入图像。为此，我们提出SAIL-Recon，一种基于前馈Transformer的大规模SfM方法，通过在场景回归网络中引入视觉定位能力来克服这一局限。具体而言，先从一组锚点图像计算神经场景表征，再以该表征为条件对回归网络进行微调，以重建所有输入图像。大量实验表明，该方法不仅可高效扩展至大规模场景，且在相机位姿估计和新视角合成基准（包括TUM-RGBD、CO3Dv2和Tanks & Temples）上均取得当前最佳性能。代码与模型已公开发布于https://hkust-sail.github.io/sail-recon/。
原文摘要 · Abstract (English)
Scene regression methods, such as VGGT, solve the Structure-from-Motion (SfM) problem by directly regressing camera poses and 3D scene structures from input images. They demonstrate impressive performance in handling images under extreme viewpoint changes. However, these methods struggle to handle a large number of input images. To address this problem, we introduce SAIL-Recon, a feed-forward Transformer for large scale SfM, by augmenting the scene regression network with visual localization capabilities. Specifically, our method first computes a neural scene representation from a subset of anchor images. The regression network is then fine-tuned to reconstruct all input images conditioned on this neural scene representation. Comprehensive experiments show that our method not only scales efficiently to large-scale scenes, but also achieves state-of-the-art results on both camera pose estimation and novel view synthesis benchmarks, including TUM-RGBD, CO3Dv2, and Tanks & Temples. We will publish our model and code. Code and models are publicly available at: https://hkust-sail.github.io/ sail-recon/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。