让多智能体学会保留次优动作，快速适应变化的最优策略。
Retaining Suboptimal Actions to Follow Shifting Optima in Multi-Agent Reinforcement Learning
- 通过学习多个次优价值函数，保留高价值备选动作
- 在复杂博弈中性能超越主流MARL算法，提升适应能力
- 适合需要动态调整策略的多智能体协作场景
价值分解是合作式多智能体强化学习的核心方法。然而，现有方法通常依赖单一最优动作，在训练过程中当基础价值函数发生改变时难以适应，常收敛至次优策略。为此，我们提出逐级次价值Q学习（S2Q），通过学习多个次价值函数来保留替代的高价值动作。将这些次价值函数融入基于Softmax的行为策略中，S2Q能持续探索，并使$Q^{ ext{tot}}$快速响应变化的最优解。在多个具有挑战性的MARL基准测试中，S2Q表现稳定优于多种MARL算法，验证了其更强的适应性与整体性能。代码已开源：https://github.com/hyeon1996/S2Q。
原文摘要 · Abstract (English)
Value decomposition is a core approach for cooperative multi-agent reinforcement learning (MARL). However, existing methods still rely on a single optimal action and struggle to adapt when the underlying value function shifts during training, often converging to suboptimal policies. To address this limitation, we propose Successive Sub-value Q-learning (S2Q), which learns multiple sub-value functions to retain alternative high-value actions. Incorporating these sub-value functions into a Softmax-based behavior policy, S2Q encourages persistent exploration and enables $Q^{\text{tot}}$ to adjust quickly to the changing optima. Experiments on challenging MARL benchmarks confirm that S2Q consistently outperforms various MARL algorithms, demonstrating improved adaptability and overall performance. Our code is available at https://github.com/hyeon1996/S2Q.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。