针对大模型探索失效问题，提出信任区域熵方法提升生成质量
TRE: Encouraging Exploration in the Trust Region
- 在模型信任区域内限制探索范围，避免无效词分布稀释
- 在数学推理、组合搜索等任务上显著优于传统方法
- 适合需要高质量长文本生成的强化学习场景
熵正则化是强化学习中增强探索的标准技术，但在拥有庞大词表和长生成序列的大语言模型（LLMs）中效果微弱甚至恶化。我们将其归因于大模型固有的累积尾部风险：标准全局熵最大化会将概率质量无差别地稀释到大量无效词的尾部，破坏连贯推理。为此，我们提出信任区域熵（TRE），仅在模型信任区域内鼓励探索。在数学推理（MATH）、组合搜索（Countdown）和偏好对齐（HH）任务上的大量实验表明，TRE始终优于原始PPO、标准熵正则化及其他探索基线。代码已开源。
原文摘要 · Abstract (English)
Entropy regularization is a standard technique in reinforcement learning (RL) to enhance exploration, yet it yields negligible effects or even degrades performance in Large Language Models (LLMs). We attribute this failure to the cumulative tail risk inherent to LLMs with massive vocabularies and long generation horizons. In such environments, standard global entropy maximization indiscriminately dilutes probability mass into the vast tail of invalid tokens rather than focusing on plausible candidates, thereby disrupting coherent reasoning. To address this, we propose Trust Region Entropy (TRE), a method that encourages exploration strictly within the model's trust region. Extensive experiments across mathematical reasoning (MATH), combinatorial search (Countdown), and preference alignment (HH) tasks demonstrate that TRE consistently outperforms vanilla PPO, standard entropy regularization, and other exploration baselines. Our code is available at https://github.com/WhyChaos/TRE-Encouraging-Exploration-in-the-Trust-Region.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。