arXiv:2502.05857cs.CVcs.AI2025-02ICCV被引 8

统一建模第一人称视角下的感知、预测与行动,提升智能体行为真实性

EgoAgent: A Joint Predictive Agent Model in Egocentric Worlds

  • 用单个Transformer联合建模感知、预测与行动,模拟人类学习循环
  • 在多个任务上超越现有方法,尤其在未来状态预测上表现更优
  • 适合研究第一人称视觉、具身智能与多任务协同的学者参考

学习一个能像人类一样,从第一人称视角同时感知环境、预测未来并采取行动的智能体模型,是计算机视觉中的基本挑战。现有方法通常分别训练这些能力的独立模型,无法捕捉其内在关联,也阻碍了彼此间的相互学习。受人类通过感知-行动循环学习的启发,我们提出EgoAgent,一个统一的智能体模型,通过单个Transformer同步学习表征、预测与行动。该模型将任务表述为状态与动作的交错序列,显式建模三者间的因果与时序依赖。进一步引入联合嵌入-动作-预测架构,采用时序非对称的预测器与观察器分支,实现三大能力的协同优化。在图像分类、第一人称未来状态预测及3D人体动作预测等代表性任务上的全面评估表明,本方法具有显著优势。代码与训练模型将公开于https://github.com/zju3dv/EgoAgent。

原文摘要 · Abstract (English)

Learning an agent model that behaves like humans-capable of jointly perceiving the environment, predicting the future, and taking actions from a first-person perspective-is a fundamental challenge in computer vision. Existing methods typically train separate models for these abilities, which fail to capture their intrinsic relationships and prevent them from learning from each other. Inspired by how humans learn through the perception-action loop, we propose EgoAgent, a unified agent model that simultaneously learns to represent, predict, and act within a single transformer. EgoAgent explicitly models the causal and temporal dependencies among these abilities by formulating the task as an interleaved sequence of states and actions. It further introduces a joint embedding-action-prediction architecture with temporally asymmetric predictor and observer branches, enabling synergistic optimization across all three capabilities. Comprehensive evaluations of EgoAgent on representative tasks such as image classification, egocentric future state prediction, and 3D human motion prediction demonstrate the superiority of our method. The code and trained models will be publicly available at https://github.com/zju3dv/EgoAgent.

第一人称视觉智能体建模多任务学习时空建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。