简单策略梯度方法在不完美信息博弈中表现优于复杂算法
Reevaluating Policy Gradient Methods for Imperfect-Information Games
- 用PPO等通用策略梯度方法替代传统复杂算法
- 7000次训练中，传统方法均未超越通用方法
- 首次公开5个大型游戏的精确可被利用性计算
过去十年，由于朴素自对弈深度强化学习（DRL）在对抗性不完美信息博弈中表现不佳，研究者发展了基于虚设演进（FP）、双盲选择（DO）和反事实遗憾最小化（CFR）的多种DRL算法。鉴于最近磁性镜面下降算法的结果，我们假设更简单的通用策略梯度方法（如PPO）在性能上可与甚至优于这些基于FP、DO、CFR的方法。为验证该假设，我们实现了首个广泛可用的五个大型博弈的精确可被利用性计算，并在此基础上开展了不完美信息博弈DRL算法史上规模最大的可被利用性对比实验。在超过7000次训练运行中，基于FP、DO、CFR的算法均未能超越通用策略梯度方法。代码已开源：https://github.com/nathanlct/IIG-RL-Benchmark 和 https://github.com/gabrfarina/exp-a-spiel。
原文摘要 · Abstract (English)
In the past decade, motivated by the putative failure of naive self-play deep reinforcement learning (DRL) in adversarial imperfect-information games, researchers have developed numerous DRL algorithms based on fictitious play (FP), double oracle (DO), and counterfactual regret minimization (CFR). In light of recent results of the magnetic mirror descent algorithm, we hypothesize that simpler generic policy gradient methods like PPO are competitive with or superior to these FP-, DO-, and CFR-based DRL approaches. To facilitate the resolution of this hypothesis, we implement and release the first broadly accessible exact exploitability computations for five large games. Using these games, we conduct the largest-ever exploitability comparison of DRL algorithms for imperfect-information games. Over 7000 training runs, we find that FP-, DO-, and CFR-based approaches fail to outperform generic policy gradient methods. Code is available at https://github.com/nathanlct/IIG-RL-Benchmark and https://github.com/gabrfarina/exp-a-spiel .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。