针对大模型强化学习中熵控制失效问题，提出自适应熵奖励机制提升探索效率。
On Entropy Control in LLM-RL Algorithms
- 设计截断熵奖励，在压缩后的词元空间计算熵以增强探索
- 自动调节熵系数，有效缓解熵带来的偏差并保留其优势
- 在多个数学推理任务中显著优于基线方法，适配不同模型与数据集
在强化学习中，恰当的熵控制对算法性能至关重要。传统方法采用熵正则化，广泛应用于PPO、SAC和A3C等算法。然而，在大语言模型强化学习（LLM-RL）中，该方法效果微弱甚至无效。本文指出，这是由于大模型响应空间过大及最优输出稀疏所致。为此，我们提出AEnt，一种基于自动调整系数的截断熵奖励机制。该方法在重归一化的较小词元空间上计算截断熵，鼓励在更紧凑的响应集中探索；同时根据截断熵值动态调整熵系数，有效控制熵引入的偏差，同时保留其益处。AEnt在不同基础模型和数据集的数学推理任务中进行测试，结果表明其在多个基准上持续优于基线方法。
原文摘要 · Abstract (English)
For RL algorithms, appropriate entropy control is crucial to their effectiveness. To control the policy entropy, a commonly used method is entropy regularization, which is adopted in various popular RL algorithms including PPO, SAC and A3C. Although entropy regularization proves effective in robotic and games RL conventionally, studies found that it gives weak to no gains in LLM-RL training. In this work, we study the issues of entropy bonus in LLM-RL setting. Specifically, we first argue that the conventional entropy regularization suffers from the LLM's extremely large response space and the sparsity of the optimal outputs. As a remedy, we propose AEnt, an entropy control method that utilizes a new clamped entropy bonus with an automatically adjusted coefficient. The clamped entropy is evaluated with the re-normalized policy defined on certain smaller token space, which encourages exploration within a more compact response set. In addition, the algorithm automatically adjusts entropy coefficient according to the clamped entropy value, effectively controlling the entropy-induced bias while leveraging the entropy's benefits. AEnt is tested in math-reasoning tasks under different base models and datasets, and it is observed that AEnt outperforms the baselines consistently across multiple benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。