打破单调性限制，让多智能体强化学习更高效可靠。
Beyond Monotonicity: Revisiting Factorization Principles in Multi-Agent Q-Learning
- 用连续梯度流建模学习动态，分析非单调分解的稳定性。
- 非单调方法在多个基准上超越单调基线，稳定收敛到最优解。
- 适合研究值分解机制或设计新MARL算法的研究者参考。
值分解是多智能体强化学习中的核心方法，通过将全局价值函数分解为局部价值，实现集中训练、分散执行。为保证个体-全局最大（IGM）一致性，现有方法要么施加单调性约束（降低表达能力），要么使用更松的代理（增加算法复杂度）。本文从动力系统角度分析非单调值分解，将学习动态建模为连续时间梯度流。证明在近似贪婪探索下，所有违反IGM一致性的零损失平衡点均为不稳定的鞍点，而仅IGM一致解是学习动态的稳定吸引子。在合成矩阵博弈和挑战性MARL基准上的大量实验表明，无约束的非单调分解能可靠恢复IGM最优解，并持续优于单调基线。此外，我们还研究了时序差分目标与探索策略的影响，为未来基于值的MARL算法设计提供实用洞见。
原文摘要 · Abstract (English)
Value decomposition is a central approach in multi-agent reinforcement learning (MARL), enabling centralized training with decentralized execution by factorizing the global value function into local values. To ensure individual-global-max (IGM) consistency, existing methods either enforce monotonicity constraints, which limit expressive power, or adopt softer surrogates at the cost of algorithmic complexity. In this work, we present a dynamical systems analysis of non-monotonic value decomposition, modeling learning dynamics as continuous-time gradient flow. We prove that, under approximately greedy exploration, all zero-loss equilibria violating IGM consistency are unstable saddle points, while only IGM-consistent solutions are stable attractors of the learning dynamics. Extensive experiments on both synthetic matrix games and challenging MARL benchmarks demonstrate that unconstrained, non-monotonic factorization reliably recovers IGM-optimal solutions and consistently outperforms monotonic baselines. Additionally, we investigate the influence of temporal-difference targets and exploration strategies, providing actionable insights for the design of future value-based MARL algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。