arXiv:2510.05251cs.CLcs.LG2025-10被引 9

通过渐变温度采样,让大模型先探索后收敛,提升推理训练效率。

Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learning

  • 生成初期用高温采样激发多样性,后期降温保证质量。
  • 在多种算法和模型上显著提升样本效率,超越固定温度策略。
  • 无需修改模型,可直接插入现有训练流程,适合强化学习推理场景。

基于可验证奖励的强化学习(RLVR)是增强大语言模型推理能力的有效方法,但其成功依赖于有效的探索策略。理想探索需在保持样本质量与确保训练稳定性之间取得平衡。标准固定温度采样难以兼顾二者:高温降低样本质量,低温限制探索范围。本文提出一种更简单有效的方法——探索性退火解码(EAD),核心思想是:探索对序列早期词元影响最大,决定语义方向。EAD在生成过程中将采样温度从高到低动态下降,实现‘前期探索、后期利用’的策略。该机制在初始阶段促进高层次多样性,后期逐步降低温度以维持样本质量,并使采样分布贴近目标策略,保障训练稳定。实验表明,EAD是一种轻量级、即插即用的方法,在多种RLVR算法和模型规模下均显著提升样本效率,优于固定温度采样。本工作表明,将探索与序列生成的自然动态对齐,为提升大模型推理能力提供了一条稳健路径。

原文摘要 · Abstract (English)

Reinforcement learning with verifiable rewards (RLVR) is a powerful paradigm for enhancing the reasoning capabilities of large language models (LLMs), yet its success hinges on effective exploration. An ideal exploration strategy must navigate two fundamental challenges: it must preserve sample quality while also ensuring training stability. While standard fixed-temperature sampling is simple, it struggles to balance these competing demands, as high temperatures degrade sample quality and low temperatures limit discovery. In this work, we propose a simpler and more effective strategy, Exploratory Annealed Decoding (EAD), grounded in the insight that exploration is most impactful on early tokens which define a sequence's semantic direction. EAD implements an intuitive **explore-at-the-beginning, exploit-at-the-end** strategy by annealing the sampling temperature from high to low during generation. This dynamic schedule encourages meaningful, high-level diversity at the start, then gradually lowers the temperature to preserve sample quality and keep the sampling distribution close to the target policy, which is essential for stable training. We demonstrate that EAD is a lightweight, plug-and-play method that significantly improves sample efficiency, consistently outperforming fixed-temperature sampling across various RLVR algorithms and model sizes. Our work suggests that aligning exploration with the natural dynamics of sequential generation offers a robust path to improving LLM reasoning.

强化学习大模型推理采样策略

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。