提升智能体对环境的理解能力,让其更可靠地执行指令。
Environmental Understanding Vision-Language Model for Embodied Agent

- 通过微调四大核心能力增强视觉语言模型的环境感知。
- 在ALFRED任务中成功率达74.3%,比基线提升8.86%。
- 适合研究具身智能体、交互式机器人与多模态推理的开发者。
视觉语言模型(VLMs)在遵循指令的具身智能体中展现出强大的感知与推理能力,但在环境理解方面仍存在局限,常因交互失败或依赖环境元数据而失效。为此,我们提出环境理解具身智能体(EUEA)框架,微调四项核心能力:1)对象感知以识别相关物体;2)任务规划生成交互子目标;3)动作理解判断成功概率;4)目标识别判定任务完成。通过在VLM上引入EUEA技能,提升了指令执行的可靠性。进一步设计恢复步骤,利用这些技能采样替代动作纠正失败案例,并引入组相对策略优化(GRPO)阶段,修正不一致的技能预测。在ALFRED任务上,该框架显著优于行为克隆基线,平均成功率提升8.86%;恢复与GRPO阶段带来额外3.03%的增益。技能层面分析揭示了封闭与开源VLM在环境理解上的关键缺陷,并明确了有效人-环境交互所需能力。
原文摘要 · Abstract (English)
Vision-language models (VLMs) have shown strong perception and reasoning abilities for instruction-following embodied agents. However, despite these abilities and their generalization performance, they still face limitations in environmental understanding, often failing on interactions or relying on environment metadata during execution. To address this challenge, we propose a novel framework named Environmental Understanding Embodied Agent (EUEA), which fine-tunes four core skills: 1) object perception for identifying relevant objects, 2) task planning for generating interaction subgoals, 3) action understanding for judging success likelihood, and 4) goal recognition for determining goal completion. By fine-tuning VLMs with EUEA skills, our framework enables more reliable task execution for instruction-following. We further introduce a recovery step that leverages these core skills and a group relative policy optimization (GRPO) stage that refines inconsistent skill predictions. The recovery step samples alternative actions to correct failure cases, and the GRPO stage refines inconsistent skill predictions. Across ALFRED tasks, our VLM significantly outperforms a behavior-cloning baseline, achieving an 8.86% improvement in average success rate. The recovery and GRPO stages provide an additional 3.03% gain, further enhancing overall performance. Finally, our skill-level analyses reveal key limitations in the environmental understanding of closed- and open-source VLMs and identify the capabilities necessary for effective agent-environment interaction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。