提出新算法缓解多智能体协作中的过泛化问题
Mitigating Relative Over-Generalization in Multi-Agent Reinforcement Learning
- 通过迭代采样最优下一状态，优化联合策略学习
- 在多个环境中提升收敛速度与样本效率
- 适合需要高效协同的多智能体系统研究
在去中心化的多智能体强化学习中，各智能体独立学习可能导致相对过泛化（RO），即最优联合动作被次优动作压制。这会阻碍合作任务中的有效协调，因为智能体倾向于选择个体理性但整体次优的动作。为此，我们提出MaxMax Q-Learning（MMQ），通过迭代采样和评估潜在下一状态，选择具有最大Q值的状态进行学习。该方法改进了理想状态转移的近似，更贴近协作智能体的最优联合策略。我们提供了支持MMQ潜力的理论分析，并在多个易受RO影响的环境中进行了实证评估。结果表明，MMQ经常优于现有基线，在收敛性和样本效率方面表现更佳。
原文摘要 · Abstract (English)
In decentralized multi-agent reinforcement learning, agents learning in isolation can lead to relative over-generalization (RO), where optimal joint actions are undervalued in favor of suboptimal ones. This hinders effective coordination in cooperative tasks, as agents tend to choose actions that are individually rational but collectively suboptimal. To address this issue, we introduce MaxMax Q-Learning (MMQ), which employs an iterative process of sampling and evaluating potential next states, selecting those with maximal Q-values for learning. This approach refines approximations of ideal state transitions, aligning more closely with the optimal joint policy of collaborating agents. We provide theoretical analysis supporting MMQ's potential and present empirical evaluations across various environments susceptible to RO. Our results demonstrate that MMQ frequently outperforms existing baselines, exhibiting enhanced convergence and sample efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。