调整BN前向模式可显著提升离散动作强化学习性能
On BatchNorm Forward Modes in Value-Based Reinforcement Learning
- 选择批量统计或运行统计模式影响BN效果
- 在26个Atari游戏中，最优配置超越层归一化表现
- 适合关注离散动作值学习优化的研究者
批归一化（BN）在连续控制的演员-评论家方法中显著提升样本效率，但近期研究发现其在Atari上的离散动作价值学习中导致性能下降。这一现象令人意外，因离散Q网络不存在CrossQ所指出的动作输入分布不匹配问题。我们发现，针对目标网络的C51和无目标网络的PQN，在特定前向传播中切换使用批量统计或运行统计模式，可逆转性能下降。在C51中，将Bootstrap前向改为批量统计模式后，性能显著优于未归一化和层归一化基线，并在更新数据比达12时仍保持稳定。在PQN中，同时在动作选择和Bootstrap中使用批量统计，可恢复因运行统计配置失败的性能。在400M帧下覆盖26个Atari游戏，该配置的最终综合得分高于使用层归一化的PQN。结果表明，精心配置的BN可大幅提升离散动作价值学习，且其前向协议是算法设计的关键部分。
原文摘要 · Abstract (English)
Batch normalization (BN) substantially improves sample efficiency in continuous-control actor-critic methods such as CrossQ, yet recent studies report performance degradation in discrete-action value learning on Atari. These failures are surprising because discrete Q-networks lack the action-input distribution mismatch identified by CrossQ. We show for target-based C51 and target-free PQN that the simple choice between running and batch statistics at specific forward passes can reverse this degradation. In C51, switching the BN bootstrap forward to batch-statistic mode significantly improves performance over unnormalized and LayerNorm baselines and scales stably with update-to-data ratios up to 12. In PQN, using batch-statistics for both action selection and bootstrapping recovers performance from the failing running-statistic configuration. Across 26 Atari games at 400M frames, this configuration achieves a higher final aggregate score than PQN with LayerNorm. Our results show that carefully configured BN can substantially improve discrete-action value learning, and that its forward protocols are an essential part of the algorithm specification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。