arXiv:2604.13596cs.CV2026-04被引 2

解决跨视角物体分割难题,实现精准像素级匹配。

VGGT-Segmentor: Geometry-Enhanced Cross-View Segmentation

论文配图:VGGT-Segmentor: Geometry-Enhanced Cross-View Segmentation
图 1 · 摘自论文原文
  • 融合几何建模与语义分割,通过三阶段头结构提升精度
  • 在Ego-Exo4D上达67.7%和68.0%的平均交并比
  • 无需成对标注,自监督预训练模型超越多数有监督方法

跨不同第一人称与第三人称视角的实例级物体分割是视觉理解中的基础挑战,对具身智能和远程协作至关重要。由于尺度、视角和遮挡变化剧烈,直接像素级匹配难以稳定。尽管现有几何感知模型如VGGT在特征对齐方面表现良好,但在密集预测任务中仍因像素级投影漂移而失效。为此,我们提出VGGT-Segmentor(VGGT-S),将强健的几何建模与像素级语义分割统一。该框架利用VGGT的跨视角特征表示,并引入新颖的联合分割头,分三阶段完成:掩码提示融合、点引导预测与迭代掩码优化,有效将高层特征对齐转化为精确分割掩码。此外,提出单图像自监督训练策略,避免成对标注,实现强泛化能力。在Ego-Exo4D基准上,VGGT-S取得新最优性能,分别达到67.7%和68.0%的平均交并比,显著优于先前方法。值得注意的是,其无对应关系预训练模型超越多数全监督基线,验证了方法的有效性与可扩展性。代码已开源:https://github.com/buaa-colalab/VGGT-S。

原文摘要 · Abstract (English)

Instance-level object segmentation across disparate egocentric and exocentric views is a fundamental challenge in visual understanding, critical for applications in embodied AI and remote collaboration. This task is exceptionally difficult due to severe changes in scale, perspective, and occlusion, which destabilize direct pixel-level matching. While recent geometry-aware models like VGGT provide a strong foundation for feature alignment, we find they often fail at dense prediction tasks due to significant pixel-level projection drift, even when their internal object-level attention remains consistent. To bridge this gap, we introduce VGGT-Segmentor (VGGT-S), a framework that unifies robust geometric modeling with pixel-accurate semantic segmentation. VGGT-S leverages VGGT's powerful cross-view feature representation and introduces a novel Union Segmentation Head. This head operates in three stages: mask prompt fusion, point-guided prediction, and iterative mask refinement, effectively translating high-level feature alignment into a precise segmentation mask. Furthermore, we propose a single-image self-supervised training strategy that eliminates the need for paired annotations and enables strong generalization. On the Ego-Exo4D benchmark, VGGT-S sets a new state-of-the-art, achieving 67.7% and 68.0% average IoU for Ego to Exo and Exo to Ego tasks, respectively, significantly outperforming prior methods. Notably, our correspondence-free pretrained model surpasses most fully-supervised baselines, demonstrating the effectiveness and scalability of our approach. Code is publicly available at: https://github.com/buaa-colalab/VGGT-S.

跨视角分割几何建模自监督学习实例分割

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。