通过随机保留边界外信号,显著提升大模型强化学习的训练稳定性。
Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals

- 在裁剪边界附近引入随机保留机制,避免重要信号丢失。
- 7B到30B模型上实验表明,性能优于DAPO和GSPO等强基线。
- 无需修改架构,可直接接入现有强化学习训练流程。
强化学习结合可验证奖励(RLVR)已成为扩展大语言模型推理能力的核心范式,但其优化常面临训练不稳定与收敛不佳的问题。通过对基于裁剪的GRPO类目标函数的系统分析,我们发现硬裁剪导致的刚性决策是当前RLVR设置中的关键瓶颈。具体而言,位于裁剪阈值附近的非边界区域可能包含重要信号,却被标准硬裁剪规则丢弃。一旦精准识别此瓶颈,仅在边界处施加简单随机扰动即可恢复显著性能增益。基于此,我们提出近边界随机救援(NSR),一种轻量级、即插即用的改进方法,通过随机保留略微超出边界的标记来恢复被丢弃的信号。尽管NSR可通过随机采样被视为期望层面的隐式梯度衰减,但消融实验显示其基于边界的随机救援机制始终优于确定性梯度衰减。在7B至30B模型规模及密集与MoE架构上的大量实验验证表明,作为即插即用方案,NSR显著提升训练稳定性,并持续优于DAPO与GSPO等强基线。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a central paradigm for scaling LLM reasoning, yet its optimization often suffers from training instability and suboptimal convergence. Through a systematic dissection of clipping-based GRPO-style objectives, we identify the rigid clipping decision induced by hard clipping as a key practical bottleneck in the studied RLVR setups. Specifically, our analysis suggests that informative signals can lie in the near-boundary region just beyond the clipping threshold, and are therefore discarded by the standard hard-clipping rule. Notably, once this bottleneck is precisely identified, even simple stochastic perturbations at the boundary can recover meaningful performance gains. Building on this finding, we propose Near-boundary Stochastic Rescue (NSR), a minimal, plug-and-play modification that stochastically retains these slightly out-of-bound tokens to recover lost signals. While NSR, via stochastic sampling, can be interpreted as inducing an implicit gradient decay in expectation, our ablations reveal that its stochastic, boundary-local rescue mechanism is consistently more effective than deterministic gradient decay. Validated by extensive experiments across model sizes from 7B to 30B and both dense and MoE architectures, as a plug-and-play solution, NSR substantially improves training stability and delivers consistent gains over strong baselines such as DAPO and GSPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。