提出轻量级训练方法，解决强化学习中批量归一化导致的策略不稳问题。
Mode-Dependent Rectification for Stable PPO Training
- 设计双阶段训练流程，动态修正训练与推理时的行为差异
- 在多种游戏和真实任务中提升PPO稳定性和性能表现
- 适用于批量归一化等模式依赖组件，无需修改网络结构
在视觉强化学习中，模式依赖的模块（如批量归一化、丢弃）虽常见却可能破坏在线策略优化。本文发现，在近端策略优化（PPO）中，批量归一化在训练与推理时的行为差异会导致策略错配、分布漂移和奖励崩溃。为此提出模式依赖修正（MDR），一种无需修改架构的轻量级双阶段训练方法，可有效稳定含模式依赖层的PPO训练。在程序生成游戏和真实世界补丁定位任务上的实验表明，MDR显著提升训练稳定性与性能，并可自然扩展至其他模式依赖组件。
原文摘要 · Abstract (English)
Mode-dependent architectural components (layers that behave differently during training and evaluation, such as Batch Normalization or dropout) are commonly used in visual reinforcement learning but can destabilize on-policy optimization. We show that in Proximal Policy Optimization (PPO), discrepancies between training and evaluation behavior induced by Batch Normalization lead to policy mismatch, distributional drift, and reward collapse. We propose Mode-Dependent Rectification (MDR), a lightweight dual-phase training procedure that stabilizes PPO under mode-dependent layers without architectural changes. Experiments across procedurally generated games and real-world patch-localization tasks demonstrate that MDR consistently improves stability and performance, and extends naturally to other mode-dependent layers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。