arXiv:2511.18242cs.CV2025-11被引 9

让第一视角视频推理先规划后验证,提升逻辑一致性。

EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning

  • 分两步:先生成第一人称动作计划,再用第三人称验证其合理性。
  • 在EgoBlind和EgoOrient上分别比Qwen2.5-VL-7B高7.7和4.4分。
  • 仅用5.2万样本训练,同时兼顾第一与第三人称任务泛化。

第一人称视频理解需在部分可观测和视角持续变化下进行流程推理。当前多模态大语言模型在此场景中表现不佳,常生成看似合理但视觉不一致或弱关联的回应。我们提出EgoVITA框架,将第一人称视频推理分解为结构化的‘计划-验证’流程。模型首先生成一个第一人称视角下的因果动作序列(即‘第一人称计划’),随后通过一个‘第三人称验证’阶段,基于同一视频的第三人称推理来检验该计划在时空与逻辑上的一致性,且无需第三人称视频输入。这种分解实现跨视角反馈,无需成对的首尾视角标注数据。为训练此推理过程,采用组相对策略优化(GRPO),引入两个密集奖励信号:一个引导预期动作与后续视觉观测对齐,另一个强化第三人称验证的一致性。EgoVITA在第一人称推理基准上达到领先性能,在EgoBlind上超越Qwen2.5-VL-7B达+7.7分,在EgoOrient上达+4.4分,且仅用52,000个样本训练,仍保持对第三人称视频任务的强大泛化能力。

原文摘要 · Abstract (English)

Egocentric video understanding requires procedural reasoning under partial observability and continuously shifting viewpoints. Current multimodal large language models (MLLMs) struggle with this setting, often generating plausible but visually inconsistent or weakly grounded responses. We introduce $\textbf{EgoVITA}$, a framework that decomposes egocentric video reasoning into a structured $\textit{plan-then-verify}$ process. The model first generates an $\textbf{egocentric plan}$: a causal sequence of anticipated actions from a first-person perspective. This plan is then evaluated by an $\textbf{exocentric verification}$ stage that uses third-person reasoning over the same video to verify its spatiotemporal and logical consistency, without exocentric video input. This decomposition enables cross-perspective feedback without requiring paired ego-exo supervision. To train this reasoning process, we adopt Group Relative Policy Optimization (GRPO) with two dense reward signals: one that grounds anticipated actions in subsequent visual observations and another that reinforces consistent third-person verification. $\textbf{EgoVITA}$ achieves state-of-the-art performance on egocentric reasoning benchmarks, outperforming Qwen2.5-VL-7B by $\mathbf{+7.7}$ on EgoBlind and $\mathbf{+4.4}$ on EgoOrient, while maintaining strong generalization on exocentric video tasks with only $52k$ training samples.

视频推理第一人称计划验证多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。