arXiv:2510.12693cs.AI2025-10被引 9

用先验知识+在线强化学习,让小模型也能当智能体。

ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learning

  • 分两阶段:先学环境知识,再通过强化学习提升能力。
  • 在复杂任务上比GPT-4o高19.4%,还能泛化到未见任务。
  • 适合想用小模型做智能体的开发者,尤其关注实用性和泛化性。

近期进展表明,视觉语言模型(VLMs)具备在复杂环境中感知、推理与交互的能力。然而,高性能系统依赖大型模型,部署成本高;小型模型则缺乏必要知识与技能。为此,我们提出Embodied Reasoning Agent(ERA),一个两阶段框架,融合先验知识学习与在线强化学习(RL)。第一阶段:基于三类数据提炼基础认知——轨迹增强先验(用强模型生成结构化推理补全轨迹)、环境锚定先验(提供环境内知识与对齐监督)、外部知识先验(从外部数据集迁移通用知识)。第二阶段构建在线RL流水线,针对长时序、稀疏奖励、训练不稳定等挑战,引入自总结机制、密集奖励设计与逐轮策略优化。在高阶规划(EB-ALFRED)与低阶控制(EB-Manipulation)任务上的实验表明,ERA-3B显著优于基于提示的大模型与先前训练基线,分别较GPT-4o提升8.4%和19.4%,且具备强泛化能力。总体而言,ERA为可扩展的具身智能提供了可行路径,为未来系统设计提供方法论启示。

原文摘要 · Abstract (English)

Recent advances in embodied AI highlight the potential of vision language models (VLMs) as agents capable of perception, reasoning, and interaction in complex environments. However, top-performing systems rely on large-scale models that are costly to deploy, while smaller VLMs lack the necessary knowledge and skills to succeed. To bridge this gap, we present \textit{Embodied Reasoning Agent (ERA)}, a two-stage framework that integrates prior knowledge learning and online reinforcement learning (RL). The first stage, \textit{Embodied Prior Learning}, distills foundational knowledge from three types of data: (1) Trajectory-Augmented Priors, which enrich existing trajectory data with structured reasoning generated by stronger models; (2) Environment-Anchored Priors, which provide in-environment knowledge and grounding supervision; and (3) External Knowledge Priors, which transfer general knowledge from out-of-environment datasets. In the second stage, we develop an online RL pipeline that builds on these priors to further enhance agent performance. To overcome the inherent challenges in agent RL, including long horizons, sparse rewards, and training instability, we introduce three key designs: self-summarization for context management, dense reward shaping, and turn-level policy optimization. Extensive experiments on both high-level planning (EB-ALFRED) and low-level control (EB-Manipulation) tasks demonstrate that ERA-3B surpasses both prompting-based large models and previous training-based baselines. Specifically, it achieves overall improvements of 8.4\% on EB-ALFRED and 19.4\% on EB-Manipulation over GPT-4o, and exhibits strong generalization to unseen tasks. Overall, ERA offers a practical path toward scalable embodied intelligence, providing methodological insights for future embodied AI systems.

具身智能强化学习视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。