利用3D模型空间先验，实现极端视角下图像段落的精准匹配。
SegMASt3R: Geometry Grounded Segment Matching
- 基于3D基础模型的空间理解，设计几何感知段落匹配架构。
- 在ScanNet++和Replica数据集上AUPRC提升最高达30%。
- 适用于3D实例映射与物体相对导航等下游任务。
段落匹配是计算机视觉中的重要中间任务，用于建立图像间语义或几何连贯区域的对应关系。与聚焦局部特征的关键点匹配不同，段落匹配捕捉结构化区域，对遮挡、光照变化和视角变换更具鲁棒性。本文利用3D基础模型的空间理解能力，解决宽基线段落匹配难题——即极端视角偏移场景下的匹配。我们提出一种架构，借助3D基础模型的归纳偏置，实现图像对间最大180度视角旋转下的段落匹配。大量实验表明，该方法在ScanNet++和Replica数据集上，相较当前最优方法（包括SAM2视频传播器和局部特征匹配方法），在AUPRC指标上最高提升30%。进一步验证了该模型在3D实例映射和物体相对导航等下游任务中的有效性。
原文摘要 · Abstract (English)
Segment matching is an important intermediate task in computer vision that establishes correspondences between semantically or geometrically coherent regions across images. Unlike keypoint matching, which focuses on localized features, segment matching captures structured regions, offering greater robustness to occlusions, lighting variations, and viewpoint changes. In this paper, we leverage the spatial understanding of 3D foundation models to tackle wide-baseline segment matching, a challenging setting involving extreme viewpoint shifts. We propose an architecture that uses the inductive bias of these 3D foundation models to match segments across image pairs with up to 180 degree view-point change rotation. Extensive experiments show that our approach outperforms state-of-the-art methods, including the SAM2 video propagator and local feature matching methods, by up to 30% on the AUPRC metric, on ScanNet++ and Replica datasets. We further demonstrate benefits of the proposed model on relevant downstream tasks, including 3D instance mapping and object-relative navigation. Project Page: https://segmast3r.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。