arXiv:2605.20388cs.CV2026-05

用视角轨迹预测动作,比语言更准且能提前推断。

How You Move Tells What You'll Do: Trajectory-Conditioned Egocentric Prediction

论文配图:How You Move Tells What You'll Do: Trajectory-Conditioned Egocentric Prediction
图 1 · 摘自论文原文
  • 用第一人称视角移动路径作为动作预测条件信号
  • 在多个数据集上超越视觉语言模型和结构化规划器,长时程预测优势更大
  • 无需真实轨迹即可实现高精度预测,适合无目标场景

预测第一人称视角的未来演化(如后续动作、任务完成情况、投篮是否命中)本质是不确定的:相同上下文可能对应多种合理未来。传统模型为降低误差而平均所有可能性,反而导致错误。本文发现:第一人称视角的运动轨迹蕴含了操作者意图,其精细程度远超语言,能有效引导动作预测;同时,该轨迹本身可从当前上下文中部分预测,无需测试时观测。基于此提出TrajPilot模型,从第一人称上下文生成候选轨迹,用于在动作对齐嵌入空间中引导动作预测,语言仅用于结构设计,不作为输入条件。在Ego-Exo4D atomic、Ego-Exo4D Keystep、Ego4D GoalStep、EgoPER等数据集上优于视觉语言模型与结构化规划基线,且预测时长越久优势越明显(传统方法在此处失效),即使仅使用RGB图像估计相机位姿仍保持性能。当目标信息在推理时被隐藏,同一模型仍可实现无目标预判,在Ego-Exo4D atomic上优于视觉语言模型,并扩展至EPIC-Kitchens-100和篮球投篮结果预测任务。

原文摘要 · Abstract (English)

Predicting how a person's first-person view will evolve (what action will follow, what plan completes a task, whether an in-progress shot will score) is fundamentally under-specified: the same context admits many plausible futures, and a model trained to minimize prediction error is forced to hedge or average across them, getting it wrong either way. Two findings shape our approach. First, the future camera trajectory, the path the head carves through space, lets the model commit to one of those futures: it carries the operator's intent in a form fine enough to determine how an action will unfold, substantially outperforming language as a conditioning signal. Second, this same intent makes the trajectory itself partially predictable from the context at hand, enough that trajectory need not be observed at test time to recover most of the gain. We instantiate these findings as TrajPilot, a model that predicts candidate future trajectories from egocentric context and uses them to pilot action prediction in an action-aligned embedding space where language shapes the structure but is never used as a conditioning input. TrajPilot beats VLM and structured-planner baselines on procedural planning across Ego-Exo4D atomic, Ego-Exo4D Keystep, Ego4D GoalStep, and EgoPER, with the trajectory advantage widening with horizon (exactly where prior planners collapse) and holding under RGB-only camera-pose estimation. With the goal masked at inference, the same model performs goal-free anticipation, beating VLM baselines on Ego-Exo4D atomic and extending to EPIC-Kitchens-100 and basketball shot-outcome prediction.

动作预测视角轨迹第一人称视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。