提出QFIX模型,解决多智能体强化学习中价值函数分解的表示能力不足问题。
Fixing Incomplete Value Function Decomposition for Multi-Agent Reinforcement Learning
- 引入轻量级修复层,完整表征满足IGM性质的价值函数。
- 在SMACv2和Overcooked上提升性能,优于QPLEX且训练更稳定。
- 结构简单、参数少,适合追求高效与稳定的多智能体研究者。
合作式多智能体强化学习中的价值函数分解方法通过个体效用组合生成联合价值,并使用联合目标进行训练。为保证个体效用与联合价值间动作选择的一致性,组合必须满足个体-全局最大(IGM)性质。尽管满足IGM本身不难,但现有方法(如VDN、QMIX)表示能力有限,无法表达全部IGM值;唯一无此限制的方法(QPLEX)却过于复杂。本文提出一种简洁的IGM值全类表述,自然导出QFIX——一类新型价值分解模型,通过添加一个轻量“修复”层扩展了先前模型的表示能力。我们推导出多个QFIX变体,并在两个主流多智能体框架中实现三种变体。在多个SMACv2和Overcooked环境中进行实验,结果表明:QFIX (i) 显著提升已有方法性能,(ii) 学习更稳定且优于主要竞争者QPLEX,(iii) 且采用最简最小的混合模型实现上述效果。
原文摘要 · Abstract (English)
Value function decomposition methods for cooperative multi-agent reinforcement learning compose joint values from individual per-agent utilities, and train them using a joint objective. To ensure that the action selection process between individual utilities and joint values remains consistent, it is imperative for the composition to satisfy the individual-global max (IGM) property. Although satisfying IGM itself is straightforward, most existing methods (e.g., VDN, QMIX) have limited representation capabilities and are unable to represent the full class of IGM values, and the one exception that has no such limitation (QPLEX) is unnecessarily complex. In this work, we present a simple formulation of the full class of IGM values that naturally leads to the derivation of QFIX, a novel family of value function decomposition models that expand the representation capabilities of prior models by means of a thin "fixing" layer. We derive multiple variants of QFIX, and implement three variants in two well-known multi-agent frameworks. We perform an empirical evaluation on multiple SMACv2 and Overcooked environments, which confirms that QFIX (i) succeeds in enhancing the performance of prior methods, (ii) learns more stably and performs better than its main competitor QPLEX, and (iii) achieves this while employing the simplest and smallest mixing models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。