改进强化学习中的策略方差问题，提升不完全信息博弈表现
GAE Falls Short in Imperfect-Information Self-Play Reinforcement Learning
- 用集中式价值函数替代采样动作，降低策略更新噪声
- 在斗地主和德州扑克中实现稳定强性能，优于标准PPO
- 适合需要高鲁棒性的多人对抗型博弈场景
不完全信息多智能体强化学习中，智能体需在部分可观测环境下与对手对抗，要求采用随机策略。尽管基于近端策略优化（PPO）的自对弈方法已取得良好实证效果，但其标准优势估计器——广义优势估计（GAE）因采样未来随机动作而引入额外方差。该方差在均衡自对弈中被放大，即使评估器精确也难以消除。为此，本文提出基于集中式动作价值评论者的新优势估计器——$Q$-boosting，并构建方差减少策略优化（VRPO）算法。该方法将多步采样备份替换为多步期望SARSA$(λ)$迹，在每一步计算策略期望以平均掉动作采样噪声，同时保留PPO的裁剪目标与在线策略更新。实验表明，VRPO在中等至大规模博弈（如斗地主、两人无限制德州扑克）中均表现优异，持续优于标准PPO。
原文摘要 · Abstract (English)
Competitive multi-agent reinforcement learning in imperfect-information games requires agents to act under partial observability and against adversarial opponents, necessitating stochastic policies. While self-play reinforcement learning with Proximal Policy Optimization (PPO) has achieved strong empirical success, its standard advantage estimator, generalized advantage estimation, suffers from additional variance due to the sampling of stochastic future actions. This variance is amplified in equilibrium self-play because of the stochastic nature of the equilibrium policy and persists even when the critic is exact. We address this bottleneck by introducing $Q$-boosting, a variance-reduced advantage estimator based on a centralized action-value critic, and propose Variance-Reduced Policy Optimization (VRPO), incorporating this new estimator. The algorithm replaces sampled multi-step backups with a multi-step Expected SARSA$(λ)$ trace, computing policy expectations at each step to average out action-sampling noise, while retaining PPO's clipped objective and on-policy actor updates. Empirically, VRPO consistently achieves strong performance from mid-sized to large-scale games including Dou Dizhu and Heads-Up No-Limit Texas Hold'em.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。