提出深度超松弛极小极大Q学习，提升双人零和博弈的收敛速度。
Deep SOR Minimax Q-learning for Two-player Zero-sum Game
- 用深度神经网络替代表格，适配高维状态空间。
- 证明算法有限时间收敛，实验显示比传统方法快。
- 可调松弛参数，适合强化学习研究者与博弈应用开发。
本文研究双人零和博弈问题。现有连续超松弛Q学习算法在表格情形下能降低相关Q-Bellman算子的收缩因子，加快基于值迭代的求解过程，但仅限于表格场景，未适用于通常处理现实高维状态-动作空间的函数逼近设置。针对此，我们提出一种结合深度神经网络作为函数逼近器的深度连续超松弛极小极大Q学习算法，适用于高维空间。我们证明了该算法的有限时间收敛性。通过数值实验，验证了所提方法相较于现有Q学习算法的有效性。消融实验揭示了关键超松弛参数不同取值的影响。
原文摘要 · Abstract (English)
In this work, we consider the problem of a two-player zero-sum game. In the literature, the successive over-relaxation Q-learning algorithm has been developed and implemented, and it is seen to result in a lower contraction factor for the associated Q-Bellman operator resulting in a faster value iteration-based procedure. However, this has been presented only for the tabular case and not for the setting with function approximation that typically caters to real-world high-dimensional state-action spaces. Furthermore, such settings in the case of two-player zero-sum games have not been considered. We thus propose a deep successive over-relaxation minimax Q-learning algorithm that incorporates deep neural networks as function approximators and is suitable for high-dimensional spaces. We prove the finite-time convergence of the proposed algorithm. Through numerical experiments, we show the effectiveness of the proposed method over the existing Q-learning algorithm. Our ablation studies demonstrate the effect of different values of the crucial successive over-relaxation parameter.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。