用动态平均策略替代均匀正则化，提升大规模博弈自对弈性能
EMAgnet: Parameter-Space EMA Regularization for Policy Gradient Self-Play in Large Games
- 以参数的指数移动平均作为自适应正则化目标
- 在多数环境下实现更低可被利用性，尤其在含冗余策略时表现更优
- 适合需稳定探索的大规模零和不完美信息博弈场景
近期研究证明，采用正则化策略梯度方法（如PPO）进行自对弈，可在双人零和不完美信息博弈中达到甚至超越专门的游戏理论算法。目前普遍采用均匀分布作为策略正则化目标，但该方法对所有动作一视同仁，忽略了动作的实际可行性。本文提出EMAgnet，通过将正则化目标设为前一迭代策略参数的指数移动平均（EMA），实现随策略演进而动态调整的自适应正则化。在标准双人零和基准以及引入探索挑战与大量严格劣策略的修改版基准上测试，相较于使用均匀磁化正则化的PPO自对弈（线性和幂律退火两种调度），EMAgnet在多数环境取得更低的可被利用性，并在包含严格劣策略的游戏中表现出一致性能提升。
原文摘要 · Abstract (English)
Recent work has established that regularized policy gradient methods such as PPO, when used in self-play, can match or exceed specialized game-theoretic algorithms for solving two-player zero-sum imperfect-information games. The uniform distribution has emerged as a strong policy regularization target for this purpose, but it regularizes equally toward all actions regardless of their viability. We introduce EMAgnet, which instead regularizes toward an exponential moving average (EMA) of the last-iterate policy's parameters, providing an adaptive regularization target that evolves with the agent's improving strategy. We evaluate EMAgnet on both standard two-player zero-sum benchmarks and modified benchmarks with exploration challenges and large numbers of strictly dominated strategies. Relative to PPO self-play with uniform-magnet regularization under both linear and power-law annealing schedules, EMAgnet achieves lower exploitability in the majority of tested environments, with consistent performance gains across games containing strictly dominated strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。