让通用策略跨代理共享数据,提升任务完成率42.2%。
Making Universal Policies Universal
- 用联合数据集训练扩散规划器,共享观察空间
- 跨代理训练使任务成功率提升42.2%
- 适合多智能体通用策略开发场景
构建能解决多种序列决策任务的通用智能体仍是重大挑战。本文在跨代理设置下研究该问题:各智能体共享相同观测空间但动作空间不同。方法基于通用策略框架,分两阶段学习:基于扩散模型的规划器生成观测序列,逆动力学模型为序列分配动作。提出在所有智能体轨迹组成的联合数据集上训练规划器,实现正向迁移,主要挑战在于将共享计划适配到各智能体的独特约束。在BabyAI环境中评估,涵盖不同复杂度任务,验证了跨代理正向迁移效果。进一步考察规划器对未见智能体的泛化能力,并与传统模仿学习对比。在多智能体联合数据上训练的通用策略,相较单一智能体数据训练,任务完成准确率最高提升42.20%。
原文摘要 · Abstract (English)
The development of a generalist agent capable of solving a wide range of sequential decision-making tasks remains a significant challenge. We address this problem in a cross-agent setup where agents share the same observation space but differ in their action spaces. Our approach builds on the universal policy framework, which decouples policy learning into two stages: a diffusion-based planner that generates observation sequences and an inverse dynamics model that assigns actions to these plans. We propose a method for training the planner on a joint dataset composed of trajectories from all agents. This method offers the benefit of positive transfer by pooling data from different agents, while the primary challenge lies in adapting shared plans to each agent's unique constraints. We evaluate our approach on the BabyAI environment, covering tasks of varying complexity, and demonstrate positive transfer across agents. Additionally, we examine the planner's generalisation ability to unseen agents and compare our method to traditional imitation learning approaches. By training on a pooled dataset from multiple agents, our universal policy achieves an improvement of up to $42.20\%$ in task completion accuracy compared to a policy trained on a dataset from a single agent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。