arXiv:2510.11417cs.CV2025-10NeurIPS被引 2

提升第一人称与第三人称视角的物体对应精度,尤其适合长视频场景。

Robust Ego-Exo Correspondence with Long-Term Memory

  • 设计双记忆架构与自适应特征路由,解决视角差异与遮挡问题。
  • 在EgoExo4D数据集上显著超越SAM2基线,实现新最优性能。
  • 适用于需要长期视觉记忆的智能助手与导航系统。

建立第一人称与第三人称视角间的物体级对应关系,对智能助手提供精准直观的视觉引导至关重要。然而,该任务面临极端视角变化、遮挡及小物体等挑战。现有方法多借鉴视频目标分割模型,仍难克服上述问题。近期,通用分割模型SAM 2展现出强泛化能力与优异性能,但直接应用于第一人称-第三人称对应(EEC)任务时,因无效的跨视角特征融合与有限的长期记忆容量,尤其在长视频中表现不佳。为此,我们提出基于SAM 2的新型EEC框架,引入长期记忆机制,包含双分支路由的内存-视图MoE模块,可沿通道与空间维度自适应分配专家特征权重;以及具备压缩策略的双记忆库系统,有效保留关键长期信息并去除冗余。在具有挑战性的EgoExo4D基准测试中,所提方法LM-EEC取得新最优结果,显著优于现有方法及SAM 2基线,展现出在多样场景下的强大泛化能力。代码与模型已开源。

原文摘要 · Abstract (English)

Establishing object-level correspondence between egocentric and exocentric views is essential for intelligent assistants to deliver precise and intuitive visual guidance. However, this task faces numerous challenges, including extreme viewpoint variations, occlusions, and the presence of small objects. Existing approaches usually borrow solutions from video object segmentation models, but still suffer from the aforementioned challenges. Recently, the Segment Anything Model 2 (SAM 2) has shown strong generalization capabilities and excellent performance in video object segmentation. Yet, when simply applied to the ego-exo correspondence (EEC) task, SAM 2 encounters severe difficulties due to ineffective ego-exo feature fusion and limited long-term memory capacity, especially for long videos. Addressing these problems, we propose a novel EEC framework based on SAM 2 with long-term memories by presenting a dual-memory architecture and an adaptive feature routing module inspired by Mixture-of-Experts (MoE). Compared to SAM 2, our approach features (i) a Memory-View MoE module which consists of a dual-branch routing mechanism to adaptively assign contribution weights to each expert feature along both channel and spatial dimensions, and (ii) a dual-memory bank system with a simple yet effective compression strategy to retain critical long-term information while eliminating redundancy. In the extensive experiments on the challenging EgoExo4D benchmark, our method, dubbed LM-EEC, achieves new state-of-the-art results and significantly outperforms existing methods and the SAM 2 baseline, showcasing its strong generalization across diverse scenarios. Our code and model are available at https://github.com/juneyeeHu/LM-EEC.

视觉对应长期记忆SAM2多视角理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。