用视觉语言知识生成虚拟数据，让机器人更好理解指令并适应新任务。
VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agents
- 用视觉语言模型预测未来状态和动作，生成符合逻辑的虚拟交互数据。
- 在未见任务上成功率比基线高24%以上，显著提升泛化能力。
- 适合研究通用机器人、具身智能与语言引导强化学习的开发者。
将大语言模型（LLMs）与强化学习（RL）结合，可使智能体更有效地解析语言指令完成任务。但LLMs缺乏对物理环境的直接感知，限制了其对环境动态的理解及对未见任务的泛化能力。为此，我们提出视觉语言知识引导的离线强化学习框架VLGOR，通过融合视觉与语言知识生成虚拟轨迹，丰富交互数据。VLGOR的核心思想是微调视觉语言模型，基于初始视觉观测和高层指令预测未来状态与动作，确保生成轨迹的时间连贯性和空间合理性。此外，采用反事实提示生成更多样化的轨迹用于离线强化学习训练，使智能体既能理解语言指令，又能基于视觉线索实现环境落地。在机器人操作基准测试中，VLGOR在未见任务上的表现显著优于基线方法，成功率达24%以上提升。
原文摘要 · Abstract (English)
Combining Large Language Models (LLMs) with Reinforcement Learning (RL) enables agents to interpret language instructions more effectively for task execution. However, LLMs typically lack direct perception of the physical environment, which limits their understanding of environmental dynamics and their ability to generalize to unseen tasks. To address this limitation, we propose Visual-Language Knowledge-Guided Offline Reinforcement Learning (VLGOR), a framework that integrates visual and language knowledge to generate imaginary rollouts, thereby enriching the interaction data. The core premise of VLGOR is to fine-tune a vision-language model to predict future states and actions conditioned on an initial visual observation and high-level instructions, ensuring that the generated rollouts remain temporally coherent and spatially plausible. Furthermore, we employ counterfactual prompts to produce more diverse rollouts for offline RL training, enabling the agent to acquire knowledge that facilitates following language instructions while grounding in environments based on visual cues. Experiments on robotic manipulation benchmarks demonstrate that VLGOR significantly improves performance on unseen tasks requiring novel optimal policies, achieving a success rate over 24% higher than the baseline methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。