解决第一人称与第三人称视角间物体关系理解难题,提升跨视角分割准确率。
ObjectRelator: Enabling Cross-View Object Relation Understanding Across Ego-Centric and Exo-Centric Perspectives
- 引入语言提示融合视觉掩码,增强物体定位精度。
- 通过自监督对齐实现跨视角一致性,应对外观变化挑战。
- 在Ego-Exo4D和HANDAL-X数据集上达到当前最优性能。
弥合第一人称与第三人称视角间的鸿沟是计算机视觉的长期挑战。本文聚焦新兴的Ego-Exo物体对应任务,旨在通过分割理解跨视角的物体关系。尽管已有多种分割模型,但多数仅处理单张图像,难以适应跨视角场景。近期提出的PSALM虽具备零样本能力,但在第一人称与第三人称视角剧烈变化下,仍难以准确定位与分割物体,尤其在复杂背景或物体外观显著变化时表现不佳。为此,本文提出ObjectRelator,包含两个核心模块:多模态条件融合(MCFuse)与基于自监督的跨视角物体对齐(XObjAlign)。MCFuse引入语言作为额外线索,融合视觉掩码与文本描述,提升定位准确性并避免错误关联;XObjAlign通过自监督对齐强化跨视角一致性,增强对物体外观变化的鲁棒性。大量实验表明,ObjectRelator在大规模Ego-Exo4D基准及适配后的HANDAL-X数据集上均取得当前最优效果。代码已公开于http://yuqianfu.com/ObjectRelator。
原文摘要 · Abstract (English)
Bridging the gap between ego-centric and exo-centric views has been a long-standing question in computer vision. In this paper, we focus on the emerging Ego-Exo object correspondence task, which aims to understand object relations across ego-exo perspectives through segmentation. While numerous segmentation models have been proposed, most operate on a single image (view), making them impractical for cross-view scenarios. PSALM, a recently proposed segmentation method, stands out as a notable exception with its demonstrated zero-shot ability on this task. However, due to the drastic viewpoint change between ego and exo, PSALM fails to accurately locate and segment objects, especially in complex backgrounds or when object appearances change significantly. To address these issues, we propose ObjectRelator, a novel approach featuring two key modules: Multimodal Condition Fusion (MCFuse) and SSL-based Cross-View Object Alignment (XObjAlign). MCFuse introduces language as an additional cue, integrating both visual masks and textual descriptions to improve object localization and prevent incorrect associations. XObjAlign enforces cross-view consistency through self-supervised alignment, enhancing robustness to object appearance variations. Extensive experiments demonstrate ObjectRelator's effectiveness on the large-scale Ego-Exo4D benchmark and HANDAL-X (an adapted dataset for cross-view segmentation) with state-of-the-art performance. Code is made available at: http://yuqianfu.com/ObjectRelator.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。