提出GROW框架，让视觉语言智能体在开放世界中更高效地学习多轮决策。
GROW: Aligning GRPO with State-Action Modeling for Open-World VLM Agents
- 将轨迹拆解为状态-动作样本，用局部优势比较替代全轨迹训练
- 在800多个Minecraft任务上达到当前最优性能
- 适合研究开放世界智能体与强化学习结合的学者
最近，视觉语言模型（VLM）智能体在开放世界任务中展现出良好进展，成功完成任务通常需要多轮视觉感知与动作执行。然而，现有方法仍主要依赖监督微调（SFT）和专家示范，先进的强化学习算法——群体相对策略优化（GRPO）尚未有效应用于此类多轮任务，因为标准GRPO需要完整轨迹作为训练样本，导致上下文过长且噪声大。为此，我们提出GROW，一种面向开放世界VLM智能体的强化学习框架：将收集到的轨迹分解为状态-动作样本，计算这些样本之间的优势差，而非将整个轨迹视为单一实体。我们进一步提供了代理分析，表明即使分组样本基于不同局部状态而非相同提示上下文，该目标在简化假设下仍能保留GRPO的核心相对策略优化信号。在超过800个Minecraft任务上的实验表明，本方法达到当前最优（SOTA）性能，验证了所提强化学习框架的有效性。
原文摘要 · Abstract (English)
Recently, vision-language model (VLM) agents have shown promising progress in open-world tasks, where successful task completion often requires multiple turns of visual perception and action execution. However, existing methods still rely primarily on Supervised Fine-Tuning (SFT) with expert demonstrations, while the advanced reinforcement learning (RL) algorithm, specifically Group Relative Policy Optimization (GRPO), has not been effectively employed for multi-turn RL in these tasks because standard GRPO requires full trajectories as training samples which leads to excessively long context and noise. To address this issue, we propose GROW, a RL framework for open-world VLM agents that decomposes collected trajectories into state-action samples, and computes advantages between these samples rather than treating a full trajectory as a single entity. We further provide a surrogate analysis indicating that, even though the grouped samples are conditioned on different local states rather than an identical prompt context, the objective can preserve the core relative policy optimization signal of GRPO under simplifying assumptions. Experiments on more than 800 Minecraft tasks show that our method achieves state-of-the-art (SOTA) performance, demonstrating the effectiveness of our proposed RL framework for open-world VLM agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。