用大模型当教练，让多机器人自动学会复杂协作。
CRAFT: Coaching Reinforcement Learning Autonomously using Foundation Models for Multi-Robot Coordination Tasks
- 用大语言模型分解长时序任务为子任务，实现自主规划。
- 通过视觉语言模型优化奖励函数，提升策略学习效率。
- 可在真实四足机器人上实现跨域迁移，适合多机协同研究者。
多智能体强化学习（MARL）为多智能体系统中的协调学习提供了强大框架，但应用于机器人时仍面临高维连续联合动作空间、复杂奖励设计及多智能体并发学习导致的非平稳性等挑战。人类学习复杂协作时常借助教练，通过精心设计的课程和详细反馈进行引导。受此启发，我们提出CRAFT：一种利用基础模型自主担任教练的多机器人协调学习框架。CRAFT利用大语言模型（LLM）的规划能力，自动将长时序协调任务分解为一系列子任务；随后，使用LLM生成的奖励函数训练每个子任务，并通过视觉语言模型（VLM）引导的奖励精炼循环进一步优化。我们在多四足机器人导航和双臂操作任务上评估了CRAFT，验证了其学习复杂协调行为的能力。此外，在多四足导航场景中，我们展示了所学策略可成功迁移到真实世界。项目主页：https://iconlab.negarmehr.com/CRAFT/
原文摘要 · Abstract (English)
Multi-Agent Reinforcement Learning (MARL) provides a powerful framework for learning coordination in multi-agent systems. However, applying MARL to robotics remains challenging due to their high-dimensional continuous joint action spaces, complex reward design, and non-stationarity from concurrently learning agents. On the other hand, humans often learn complex coordination with the help of coaches, who guide learning through carefully designed curricula and detailed feedback. Building on the reasoning capabilities of foundation models, we argue that these models can similarly coach robots to learn coordination. Motivated by this, we propose CRAFT: Coaching Reinforcement learning Autonomously using Foundation models for learning coordination Tasks, a framework that leverages foundation models to act as a "coach" for multi-robot coordination. CRAFT automatically decomposes long-horizon coordination tasks into sequences of subtasks using the planning capability of Large Language Models (LLMs). Then, CRAFT trains each subtask using LLM-generated reward functions, and refines them through a Vision Language Model (VLM)-guided reward-refinement loop. We evaluate CRAFT on multi-quadruped navigation and bimanual manipulation tasks, and demonstrate its capability to learn complex coordination behaviors. In addition, in a multi-quadruped navigation setting, we show that our learned policies transfer to the real world. Project website is https://iconlab.negarmehr.com/CRAFT/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。