用自监督学习让VGGT模型在无标签场景下实现大规模定位,提升几何感知能力。
GPA-VGGT:Adapting VGGT to Large Scale Localization by Self-Supervised Learning with Geometry and Physics Aware Loss
- 通过序列级几何约束替代成对关系,增强时间特征一致性。
- 联合优化物理光照与几何约束,在无真值条件下训练模型。
- 适合大规模视觉定位、无标注数据场景下的应用者参考。
基于Transformer的通用视觉几何框架在相机位姿估计和3D场景理解中表现优异。近期的视觉几何锚定变压器(VGGT)模型在相机位姿估计和3D重建方面展现出巨大潜力。然而,这些模型通常依赖真实标签进行训练,在适应无标签和未见场景时面临挑战。本文提出一种自监督框架,利用无标签数据训练VGGT,从而提升其在大规模环境中的定位能力。为此,我们将传统的成对关系扩展为序列级几何约束:在每个序列中,采样多个源帧并将其几何投影到不同目标帧,以改善时间特征一致性。我们构建了物理光照一致性与几何约束的联合优化损失,避免对硬标签的依赖。通过该方法训练后,局部与全局跨视图注意力层以及相机和深度头均能有效捕捉多视图几何结构。实验表明,模型仅需数百次迭代即可收敛,并在大规模定位任务中取得显著提升。代码将发布于 https://github.com/X-yangfan/GPA-VGGT。
原文摘要 · Abstract (English)
Transformer-based general visual geometry frameworks have shown promising performance in camera pose estimation and 3D scene understanding. Recent advancements in Visual Geometry Grounded Transformer (VGGT) models have shown great promise in camera pose estimation and 3D reconstruction. However, these models typically rely on ground truth labels for training, posing challenges when adapting to unlabeled and unseen scenes. In this paper, we propose a self-supervised framework to train VGGT with unlabeled data, thereby enhancing its localization capability in large-scale environments. To achieve this, we extend conventional pair-wise relations to sequence-wise geometric constraints for self-supervised learning. Specifically, in each sequence, we sample multiple source frames and geometrically project them onto different target frames, which improves temporal feature consistency. We formulate physical photometric consistency and geometric constraints as a joint optimization loss to circumvent the requirement for hard labels. By training the model with this proposed method, not only the local and global cross-view attention layers but also the camera and depth heads can effectively capture the underlying multi-view geometry. Experiments demonstrate that the model converges within hundreds of iterations and achieves significant improvements in large-scale localization. Our code will be released at https://github.com/X-yangfan/GPA-VGGT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。