让模型从第三人称视频中推断第一人称交互，无需佩戴摄像头。
From My View to Yours: Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervision
- 用同步的第一、第三人称视频对训练，将第一视角作为训练时的额外监督信号。
- 在10个任务上超越现有方法，在ADL-X基准上达到新纪录。
- 适合需要理解人类动作交互但无法部署穿戴设备的场景。
视觉语言模型（VLM）在众多视频理解任务中表现优异，但其视角不变性训练限制了从第三人称视频中推断第一人称属性（如人物交互）的能力。这一缺陷在日常生活活动（ADL）监测等应用中尤为关键，因第一人称信息至关重要，但穿戴式第一人称摄像头部署常不现实。本文提出Ego2ExoVLM框架，通过训练阶段使用时间同步的头戴-第三人称视频对，使模型能直接从第三人称视频中推断第一人称属性。核心思路是将第一人称视角视为训练期间的特权监督，提供仅训练时可用的丰富交互信号。Ego2ExoVLM包含两个互补组件：Ego2Exo序列蒸馏，通过语言级序列蒸馏转移第一人称推理能力；以及Ego自适应视觉标记，促使模型在第三人称视觉表示中突出与交互相关的关键线索。为评估该能力，我们引入Ego-in-Exo Perception基准，用于评测从第三人称视频中理解第一人称属性的能力。我们在10项任务上评估Ego2ExoVLM，涵盖Ego-in-Exo Perception和现有ADL基准，结果在ADL-X基准套件上达到最先进水平，并在新提出的基准上持续优于多个强基线。所有代码、模型和数据将开源至https://github.com/dominickrei/EgoExo4ADL。
原文摘要 · Abstract (English)
Vision Language Models (VLMs) have achieved strong performance across a wide range of video understanding tasks. However, their viewpoint-invariant training limits their ability to infer egocentric properties, such as human-object interactions, from exocentric video observations. This limitation is particularly critical for applications such as Activities of Daily Living (ADL) monitoring, where understanding egocentric properties is essential but deploying wearable egocentric cameras is often impractical. We propose Ego2ExoVLM, a framework that enables VLMs to infer egocentric properties directly from exocentric videos by leveraging time-synchronized ego-exo video pairs during training. Our key insight is to treat the egocentric viewpoint as privileged supervision, providing rich interaction signals that are available only during training. Ego2ExoVLM consists of two complementary components: Ego2Exo Sequence Distillation, which transfers egocentric reasoning through a language-level sequence distillation objective, and Ego Adaptive Visual Tokens, which encourage the model to surface interaction-relevant cues within exocentric visual representations. To evaluate this capability, we introduce Ego-in-Exo Perception, a benchmark for assessing the understanding of egocentric properties from exocentric videos. We evaluate Ego2ExoVLM on 10 tasks spanning Ego-in-Exo Perception and existing ADL benchmarks, achieving state-of-the-art performance on the ADL-X benchmark suite and consistently outperforming strong baselines on our proposed benchmark. All code, models, and data will be released at https://github.com/dominickrei/EgoExo4ADL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。