针对大模型推理优化,按熵值区分令牌更新强度,提升推理能力。
Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR
- 根据令牌熵值动态调节更新强度,保留生成顺序依赖
- 在数学与代码任务上显著提升通过率,最高提升12.3%
- 适合需要精细优化推理能力的LLM研究者使用
强化学习结合可验证奖励(RLVR)已成为提升大语言模型推理能力的有效后训练方法。然而,现有方法对所有令牌施加统一优化约束,忽略了其角色差异。研究表明,高熵令牌与推理密切相关,低熵令牌主要编码事实知识。近期方法尝试通过掩码或异步训练分离令牌更新,但我们认为这种隔离破坏了自回归生成的序列依赖结构,导致学习效果不佳。为此,我们提出Archer框架,一种基于熵的双令牌约束方法,在保持联合优化的同时,调节不同令牌类型的更新强度。该方法引入响应级熵归一化以稳定令牌分类,并对推理令牌采用更宽松的裁剪范围和KL正则化,鼓励探索,同时保护知识令牌。在数学推理和代码生成基准测试中,Archer在多个模型规模下持续优于强基线,显著提升pass@1和pass@K指标。结果表明,在设计细粒度强化学习优化策略时,尊重序列级依赖至关重要。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has become an effective post-training method for improving the reasoning abilities of Large Language Models (LLMs). However, existing methods mainly apply uniform optimization constraints across all tokens, ignoring their heterogeneous roles. Prior work shows that high-entropy tokens are closely tied to reasoning, while low-entropy tokens primarily encode factual knowledge, and recent approaches attempt to exploit this distinction by isolating token updates via masking or asynchronous training. We argue that such isolation breaks the sequential dependency structure of autoregressive generation, leading to suboptimal learning. To address this, we propose \textbf{Archer}, an entropy-aware RLVR framework with \textbf{dual-token constraints} that preserves joint optimization while modulating update strength across token types. Our method introduces response-level entropy normalization for stable token classification and applies differentiated clipping ranges and KL regularization to encourage exploration on reasoning tokens while preserving knowledge tokens. Experiments on mathematical reasoning and code generation benchmarks show that Archer consistently outperforms strong baselines across multiple model scales, improving both \textit{pass@1} and \textit{pass@K} performance. These results highlight the importance of respecting sequence-level dependencies when designing fine-grained RL optimization strategies for LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。