让大模型与视觉模型实时协作,提升机器人规划能力。
EMAC+: Embodied Multimodal Agent for Collaborative Planning with VLM+LLM
- 用双向训练让大模型和视觉模型动态协作,实时优化计划。
- 在ALFWorld和RT-1上表现更优,抗干扰强,学习效率高。
- 适合研究多模态智能体、具身智能与机器人协同规划的学者。
尽管大语言模型(LLM)在文本推理与规划任务中表现优异,但在机器人控制中仍受限于三大缺陷:(1)主要处理文本输入,难以理解视觉信息;(2)现有多模态代理将LLM视为静态规划器,脱离环境动态,导致行动缺乏领域知识;(3)LLM无法从视觉交互中学习,难以优化特定领域的策略。本文提出EMAC+,一种具身多模态智能体,通过双向训练机制融合LLM与视觉语言模型(VLM)。不同于以往方法,EMAC+利用执行低层视觉控制的VLM实时反馈,动态修正由LLM生成的高层文本计划。该方法使LLM能直接通过交互经验内化视觉环境动态,而非依赖静态符号映射。在ALFWorld和RT-1基准上的大量实验表明,EMAC+在任务性能、抗噪声观测能力及学习效率方面均显著优于基线。我们还进行了详尽的消融研究,并分析成功与失败案例。
原文摘要 · Abstract (English)
Although LLMs demonstrate proficiency in several text-based reasoning and planning tasks, their implementation in robotics control is constrained by significant deficiencies: (1) LLM agents are designed to work mainly with textual inputs rather than visual conditions; (2) Current multimodal agents treat LLMs as static planners, which separates their reasoning from environment dynamics, resulting in actions that do not take domain-specific knowledge into account; and (3) LLMs are not designed to learn from visual interactions, which makes it harder for them to make better policies for specific domains. In this paper, we introduce EMAC+, an Embodied Multimodal Agent that collaboratively integrates LLM and VLM via a bidirectional training paradigm. Unlike existing methods, EMAC+ dynamically refines high-level textual plans generated by an LLM using real-time feedback from a VLM executing low-level visual control tasks. We address critical limitations of previous models by enabling the LLM to internalize visual environment dynamics directly through interactive experience, rather than relying solely on static symbolic mappings. Extensive experimental evaluations on ALFWorld and RT-1 benchmarks demonstrate that EMAC+ achieves superior task performance, robustness against noisy observations, and efficient learning. We also conduct thorough ablation studies and provide detailed analyses of success and failure cases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。