提出首个基于学习的长时第一视角视频物体追踪方法，解决物体离开视野仍需持续追踪的问题。
Whareformer: Learning to Track What is Where in Long Egocentric Videos
- 基于Transformer设计双组件模型，融合外观与3D位置信息，动态更新轨迹记忆。
- 仅用56段视频训练，在3个数据集共260段长视频上达到最优性能。
- 适合长期跟踪、遮挡严重的第一视角视频场景，如厨房操作分析。
最近提出的‘视线之外，不在记忆之外’（OSNOM）任务关注第一视角视频中佩戴者移动的物体，要求在线追踪这些物体，即使其离开视域或严重遮挡也需持续保持对实例位置的记忆。本文提出首个基于学习的OSNOM解决方案：Whareformer，一种基于Transformer的模型，包含可更新的轨迹记忆和前馈式轨迹分配模块。该模型联合推理物体外观（what）与更新后的3D位置（where），并使用专用的新轨迹标记来识别新出现物体。得益于采用相对距离和动态轨迹表示的设计，模型仅在56段视频上训练，便在三个数据集（EPIC-KITCHENS-100未见视频、IT3DEgo、HD-EPIC）共260段长视频上取得最佳表现，显著优于先前方法。
原文摘要 · Abstract (English)
The recently established 'Out of Sight, Not out of Mind' (OSNOM) task for egocentric videos focuses on tracking objects that are moved by the camera wearer, online, maintaining knowledge of instance locations throughout the video even when they leave the field of view or become heavily occluded. In this paper, we propose the first learning-based solution to the OSNOM task: Whareformer, a transformer-based model with two components: an updatable memory of established tracks and a track assignment module that associates observations with existing tracks in a feed-forward manner. Whareformer jointly reasons over evolving object appearance (what) and updated 3D location (where), and employs a dedicated New Track token to reason about novel objects. Thanks to its design choices of using relative distances and evolving track representations, Whareformer is trained on a small set of 56 videos but achieves SOTA performance on 260 long test videos from three datasets: EPIC-KITCHENS-100 (unseen videos), IT3DEgo, and HD-EPIC, with significant absolute improvements over prior work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。