提出保持策略熵的强化学习方法，提升模型探索多样性与泛化能力。
Entropy-Preserving Reinforcement Learning
- 通过分析主流策略梯度算法对熵的影响，发现其训练中熵会自然下降。
- 设计REPO与ADAPO算法，显式控制优势函数以维持探索多样性。
- 训练后模型更健壮，适合在新环境持续学习，尤其适用于复杂推理任务。
策略梯度算法推动了语言模型推理的多项进展，其关键优势在于能从自身轨迹中自主探索，促进多样化和创造性解法。然而本文揭示，多数策略梯度算法在训练过程中会自然降低策略熵，从而限制探索能力。我们主张应全程主动监控并控制熵。通过形式化分析主流策略梯度目标对熵动态的影响，识别出数值精度等关键因素对熵行为的重大影响，并提出显式熵控制机制。包括一类修改优势函数的算法家族REPO，以及自适应非对称裁剪方法ADAPO。采用熵保持方法训练的模型在整个训练过程中维持探索多样性，最终策略性能更强，且在新环境中仍具备持续学习能力。
原文摘要 · Abstract (English)
Policy gradient algorithms have driven many recent advancements in language model reasoning. An appealing property is their ability to learn from exploration on their own trajectories, a process crucial for fostering diverse and creative solutions. As we show in this paper, many policy gradient algorithms naturally reduce the entropy -- and thus the diversity of explored trajectories -- as part of training, yielding a policy increasingly limited in its ability to explore. In this paper, we argue that entropy should be actively monitored and controlled throughout training. We formally analyze the contributions of leading policy gradient objectives on entropy dynamics, identify empirical factors (such as numerical precision) that significantly impact entropy behavior, and propose explicit mechanisms for entropy control. These include REPO, a family of algorithms that modify the advantage function to regulate entropy, and ADAPO, an adaptive asymmetric clipping approach. Models trained with our entropy-preserving methods maintain diversity throughout training, yielding final policies that are more performant and retain their trainability for sequential learning in new environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。