通过去除干扰词提升大模型推理效率，训练更快更稳定。
Less Noise, More Voice: Reinforcement Learning for Reasoning via Instruction Purification
- 识别并移除提示中的干扰词，先净化再训练。
- 数学推理任务上性能提升3.88%，速度加快1.6倍。
- 适合需要高效强化学习推理的科研与工程场景。
基于可验证奖励的强化学习（RLVR）虽推动了大模型推理发展，但在有限采样预算下仍面临探索效率低、采样成功率低及训练不稳的问题。我们发现，多数探索失败并非源于任务难度，而是由少数提示词引入的干扰所致。基于此，提出少噪声采样框架LENS：先识别并移除干扰词，净化提示；再将净化过程中的成功轨迹用于监督原始噪声提示下的策略优化，使模型学会在真实噪声提示中忽略干扰。实验表明，LENS显著优于GRPO，数学推理平均提升3.88%，速度超1.6倍；科学与通用推理提升1.83%。本工作揭示了剔除干扰词对提升采样效率的关键作用，为RLVR研究提供新视角。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has advanced LLM reasoning, but remains constrained by inefficient exploration under limited rollout budgets, leading to low sampling success and unstable training in complex tasks. We find that many exploration failures arise not from problem difficulty, but from a small number of prompt tokens that introduce interference. Building on this insight, we propose the Less Noise Sampling Framework (LENS), which first prompts by identifying and removing interference tokens. then transfers successful rollouts from the purification process to supervise policy optimization on the original noisy prompts, enabling the model to learn to ignore interference in the real-world, noisy prompting settings. Experimental results show that LENS significantly outperforms GRPO, delivering higher performance and faster convergence, with a 3.88% average gain and over 1.6$\times$ speedup on math reasoning, and a 1.83% gain on scientific and general reasoning. Our work highlights the critical role of pruning interference tokens in improving rollout efficiency, offering a new perspective for RLVR research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。