多视角图像实例分割匹配,提升跨视角物体对应精度与导航成功率。
MuViSeg: Multi-View Segment Correspondences from Dense Geometry Priors

- 基于多视角联合自注意力机制,实现跨视图物体对应关系推断。
- 在Replica和Virtual KITTI 2上,相比基线提升4.85和25.9 AUPRC。
- 无需重训练即可提升机器人导航任务成功率达70%,适合场景理解应用。
传统图像匹配在稀疏关键点或密集像素层面进行,但下游任务如物体级映射、拓扑导航等需处理完整物体。近期工作通过直接在实例分割层面匹配来缩小差距:使用无类别分割器对每张图像进行分割,并利用大型3D基础模型在掩码上池化特征获取每段描述符。本文在此基础上提出三种可学习的匹配头:一种采用LightGlue风格注意力机制并结合DoubleSoftmax评分的冻结MASt3R描述符头;一种基于DPT风格的多尺度融合模块,在池化前从VGGT基础模型中提取分层空间细节;以及主要贡献——多视角扩展,通过同时对多个视角的分割段进行联合自注意力,恢复仅成对匹配无法达到的传递性对应关系。在控制视角变化范围为0°至180°的分层零样本协议下,该方法在Replica上比参数无关的Sinkhorn匹配提高+4.85 AUPRC,Virtual KITTI 2上提高+25.9 AUPRC。将多视角变体部署到RoboHop拓扑导航流水线,在Habitat-Matterport 3D(HM3D)实例图像导航基准测试中,未重新训练即实现成功率从50%提升至70%;LightGlue式头使SPL从45.7提升至59.1。
原文摘要 · Abstract (English)
Classical image correspondence is solved at the level of sparse keypoints or dense pixels, but the systems that consume these matches - object-level mapping, topological navigation, scene-graph maintenance - reason about whole objects. Recent work narrows this gap by matchng directly at the level of instance segments: a class-agnostic segmenter partitions each image, and per-segment descriptors are obtained by pooling features from large 3D foundation models over the masks. We build on this segment-level matching paradigm and propose three learned matching heads: a LightGlue-style attention head with DoubleSoftmax scoring on frozen MASt3R descriptors; a DPT-style multi-scale fusion module that exposes layered spatial detail from the VGGT foundation model before pooling; and - as our main contribution - a multi-view extension that performs joint self-attention over segments drawn from several views at once, recovering transitive correspondences that strictly pairwise matchers cannot reach. Under a stratified zero-shot protocol on Replica and Virtual KITTI 2 with controlled viewpoint baselines from 0 deg to 180 deg, the LightGlue-style head improves over a parameter-free Sinkhorn matcher on the same MASt3R backbone by +4.85 AUPRC on Replica and +25.9 AUPRC on Virtual KITTI 2. Dropped into the RoboHop topological navigation pipeline on the Habitat-Matterport 3D (HM3D) Instance Image Navigation benchmark without retraining, our multi-view variant raises success rate from 50% to 70%, and our LightGlue-style head raises SPL from 45.7 to 59.1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。