发现强化学习中的裁剪机制会改变语言模型的熵，影响探索能力。
Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models
- 裁剪下限提高熵，上限降低熵，两者共同作用导致熵下降
- 即使奖励随机，标准裁剪参数仍会使熵持续减少
- 可通过调整裁剪值主动控制熵，防止训练过早收敛
基于可验证奖励的强化学习（RLVR）是提升大语言模型推理能力的主流方法。然而，该方法易出现熵坍缩现象，即模型快速趋于确定性输出，阻碍长期训练中的探索与进步。本文通过理论与实证分析发现，PPO和GRPO中的裁剪机制会对熵产生偏差：clip-low增加熵，clip-high降低熵。在标准裁剪参数下，clip-high效应占主导，导致整体熵下降，甚至在纯随机奖励下也会发生熵减少。这表明裁剪机制本身是影响推理行为的一个未被重视的混淆因素。进一步分析显示，可主动调节裁剪值来控制熵：采用更激进的clip-low值可提升熵、促进探索，从而有效防止熵坍缩。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has recently emerged as the leading approach for enhancing the reasoning capabilities of large language models (LLMs). However, RLVR is prone to entropy collapse, where the LLM quickly converges to a near-deterministic form, hindering exploration and progress during prolonged RL training. In this work, we reveal that the clipping mechanism in PPO and GRPO induces biases on entropy. Through theoretical and empirical analyses, we show that clip-low increases entropy, while clip-high decreases it. Further, under standard clipping parameters, the effect of clip-high dominates, resulting in an overall entropy reduction even when purely random rewards are provided to the RL algorithm. Our findings highlight an overlooked confounding factor in RLVR: independent of the reward signal, the clipping mechanism influences entropy, which in turn affects the reasoning behavior. Furthermore, our analysis demonstrates that clipping can be deliberately used to control entropy. Specifically, with a more aggressive clip-low value, one can increase entropy, promote exploration, and ultimately prevent entropy collapse in RLVR training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。