用扩散模型自动分解多智能体任务,提升复杂协作效率。
Conditional Diffusion Model for Multi-Agent Dynamic Task Decomposition
- 高层策略用扩散模型预测环境变化,指导子任务选择。
- 在多个基准上优于现有方法,显著降低训练样本需求。
- 适合需要动态分工的长时程多智能体协作场景。
任务分解在复杂协作式多智能体强化学习(MARL)中展现出巨大潜力,有助于在动态不确定环境中高效完成长时程任务。然而,从零开始学习动态任务分解通常需要大量训练样本,尤其在部分可观测条件下探索庞大的联合动作空间。本文提出条件扩散模型用于动态任务分解(C$\text{D}^\text{3}$T),一种两级分层MARL框架,可自动推断子任务与协作模式。高层策略学习子任务表示,并基于子任务影响生成子任务选择策略。为捕捉子任务对环境的影响,C$\text{D}^\text{3}$T利用条件扩散模型预测下一时刻观测和奖励。低层中,智能体在其分配的子任务内协作学习并共享专有技能。此外,所学子任务表示作为语义信息融入多头注意力混合网络,增强价值分解,并在个体与联合价值函数间建立高效推理桥梁。在多个基准测试中的实验结果表明,C$\text{D}^\text{3}$T性能优于现有基线。
原文摘要 · Abstract (English)
Task decomposition has shown promise in complex cooperative multi-agent reinforcement learning (MARL) tasks, which enables efficient hierarchical learning for long-horizon tasks in dynamic and uncertain environments. However, learning dynamic task decomposition from scratch generally requires a large number of training samples, especially exploring the large joint action space under partial observability. In this paper, we present the Conditional Diffusion Model for Dynamic Task Decomposition (C$\text{D}^\text{3}$T), a novel two-level hierarchical MARL framework designed to automatically infer subtask and coordination patterns. The high-level policy learns subtask representation to generate a subtask selection strategy based on subtask effects. To capture the effects of subtasks on the environment, C$\text{D}^\text{3}$T predicts the next observation and reward using a conditional diffusion model. At the low level, agents collaboratively learn and share specialized skills within their assigned subtasks. Moreover, the learned subtask representation is also used as additional semantic information in a multi-head attention mixing network to enhance value decomposition and provide an efficient reasoning bridge between individual and joint value functions. Experimental results on various benchmarks demonstrate that C$\text{D}^\text{3}$T achieves better performance than existing baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。