提出最优分词基线，显著降低长序列强化学习的训练方差。
The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL
- 基于梯度范数反比加权更新，实现最优方差控制。
- 仅需4个样本即达32个样本性能，节省超65%计算资源。
- 适合需要高效长序列推理的LLM强化学习任务。
大语言模型的强化学习在长序列任务中常因梯度方差爆炸导致训练崩溃。虽常用基线缓解，但传统值函数难优化，且标准组基线忽略序列异质性。经典最优基线理论虽能全局降方差，却忽视分词异质性且需高成本梯度计算。本文从原理推导出最优分词基线（OTB），证明梯度更新应按累积梯度范数倒数加权。为提高效率，提出仅用前向概率估算梯度范数的对数几率-梯度代理方法。该方法实现训练稳定，在单轮和工具集成推理任务中，仅用$N=4$即达到$N=32$性能，减少超过65%的分词消耗。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) for Large Language Models (LLMs) often suffers from training collapse in long-horizon tasks due to exploding gradient variance. To mitigate this, a baseline is commonly introduced for advantage computation; however, traditional value models remain difficult to optimize, and standard group-based baselines overlook sequence heterogeneity. Although classic optimal baseline theory can achieve global variance reduction, it neglects token heterogeneity and requires prohibitive gradient-based computation. In this work, we derive the Optimal Token Baseline (OTB) from first principles, proving that gradient updates should be weighted inversely to their cumulative gradient norm. To ensure efficiency, we propose the Logit-Gradient Proxy that approximates the gradient norm using only forward-pass probabilities. Our method achieves training stability and matches the performance of large group sizes ($N=32$) with only $N=4$, reducing token consumption by over 65\% across single-turn and tool-integrated reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。