通过密集物体匹配实现第一人称与第三人称视角的精准对应
DOMR: Establishing Cross-View Segmentation via Dense Object Matching
- 基于物体间位置与语义关系，联合建模多物体对应关系
- 在Ego→Exo和Exo→Ego任务上分别达到49.7%和55.2%的平均交并比
- 适用于视觉理解、人机交互等需要跨视角对齐的场景
跨视角物体对应涉及在第一人称（自视）与第三人称（他视）视角之间匹配物体，是视觉理解的关键挑战。本文提出密集物体匹配与精炼（DOMR）框架，核心为密集物体匹配器（DOM）模块，联合建模多个物体。不同于直接将单个物体掩码与图像特征匹配的方法，DOM利用物体间的空间位置与语义关系寻找对应。该模块结合候选生成与密集匹配机制，联合编码视觉、空间与语义线索，显式构建物体间关系，实现物体间的密集匹配。此外，引入掩码精炼头以提升预测掩码的完整性和准确性，构成完整的DOMR框架。在Ego-Exo4D基准上的大量评估表明，本方法在Ego→Exo任务上达到49.7%的平均交并比，在Exo→Ego任务上达55.2%，较之前方法分别提升5.8%与4.3%，验证了其在跨视角理解中的有效性。
原文摘要 · Abstract (English)
Cross-view object correspondence involves matching objects between egocentric (first-person) and exocentric (third-person) views. It is a critical yet challenging task for visual understanding. In this work, we propose the Dense Object Matching and Refinement (DOMR) framework to establish dense object correspondences across views. The framework centers around the Dense Object Matcher (DOM) module, which jointly models multiple objects. Unlike methods that directly match individual object masks to image features, DOM leverages both positional and semantic relationships among objects to find correspondences. DOM integrates a proposal generation module with a dense matching module that jointly encodes visual, spatial, and semantic cues, explicitly constructing inter-object relationships to achieve dense matching among objects. Furthermore, we combine DOM with a mask refinement head designed to improve the completeness and accuracy of the predicted masks, forming the complete DOMR framework. Extensive evaluations on the Ego-Exo4D benchmark demonstrate that our approach achieves state-of-the-art performance with a mean IoU of 49.7% on Ego$\to$Exo and 55.2% on Exo$\to$Ego. These results outperform those of previous methods by 5.8% and 4.3%, respectively, validating the effectiveness of our integrated approach for cross-view understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。