让3D重建在有干扰物的复杂场景中依然准确
Visual Geometry Transformer in the Wild: Distractor-Free 3D Reconstruction

- 用注意力机制分离干净特征与干扰物特征
- 在真实场景下实现领先精度,无需额外3D标注
- 适合实际应用中存在遮挡和动态干扰的场景
当前端到端多视角3D重建方法虽效果出色,但依赖静态理想假设:场景无干扰物且视图间几何一致。这一假设导致先进方法在真实场景中因瞬时干扰和遮挡而失效。为此,我们提出视觉几何变压器(VGTW),一种从不一致视图中实现鲁棒重建的端到端框架。核心是识别并抑制受干扰区域,同时保留跨视图的一致成分。我们引入干扰感知训练(DAT)策略,在注意力机制中分离干净特征与受干扰特征,并强制图像间特征一致性。为此,我们构建了一个新数据集,包含像素级干扰掩码,并通过辅助掩码预测头进行监督训练。最终的VGTW为前馈网络,直接输出无干扰点云。其无需额外3D监督,计算高效,且可兼容现有流程。大量实验验证了该方法在多样化真实场景中的领先性能与强泛化能力。
原文摘要 · Abstract (English)
Current end-to-end multi-view 3D reconstruction methods achieve impressive results, but rely on a restrictive static assumption: the scenes is entire distractor-free with perfect cross-view geometry. This reliance on idealized inputs causes even the most advanced methods to fail in real-world settings, where transient distractors and occlusions present. To address this, we propose Visual Geometry Transformer in the Wild (VGTW), an end-to-end framework for robust reconstruction from inconsistent views. At its core, we isolate and suppress distractor-affected regions while preserving the consistent components across views. Specifically, we introduce a Distractor-aware Training (DAT) strategy that separates clean features from distractor-contaminated ones in the attention mechanism while enforcing feature consistency across images. To enable this, we train the model with an auxiliary mask prediction head, using supervision from a new dataset we collected with pixel-level distractor masks. The resulting VGTW model is a feed-forward network that directly outputs clean, distractor-free point clouds. Remarkably, it requires no additional 3D supervision, remains computationally efficient, and is compatible with existing pipelines. Extensive experiments validate our approach, demonstrating state-of-the-art performance and robust generalization in diverse, real-world scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。