提出差分平滑方法，解决大模型强化学习后多样性下降问题。
Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning
- 基于选择与强化偏差理论，证明强化学习导致输出单调。
- 仅在正确路径上调整奖励，实现正确性与多样性的同步提升。
- 在1B到7B模型上验证，数学推理任务最高提升6.7%。
强化学习微调大型语言模型常导致输出多样性下降。现有方法多为经验性策略，常以牺牲准确性为代价换取多样性，且效果因任务而异，甚至相互矛盾。本文首次从理论上证明了多样性下降源于选择与强化偏差。关键发现是：为缓解多样性下降，只需对正确轨迹进行奖励调整。基于此，提出一种原理性方法——差分平滑，可同时提升准确性和多样性，优于标准强化学习及常用的熵正则化方法。理论分析阐明了现有启发式方法的有效边界与失效原因，表明差分平滑具有普适优势。在1B至7B参数模型上，覆盖CountDown和真实数学推理等任务的实验显示一致性能提升，尤其在AIME24数据集上Pass@1与Pass@k分别最高提升6.7%。
原文摘要 · Abstract (English)
It is widely recognized that reinforcement learning (RL) fine-tuning of large language models often leads to diversity collapse, where outputs lack variety. Prior work has proposed a range of heuristics to counteract this effect, but these methods are ad hoc: they frequently trade off correctness for diversity, their effectiveness varies across tasks, and in some cases they even contradict one another. In this work, we place these observations on a rigorous foundation. We first provide a formal proof of why RL fine-tuning exhibits diversity collapse via a selection and reinforcement bias. Next, we make a key observation that any reward modification to address diversity collapse only needs to be applied on the correct trajectories. Building directly on this analysis, we introduce a principled method -- differential smoothing -- that provably improves both correctness and diversity, outperforming vanilla RL as well as widely used entropy-based heuristics. Our theory precisely characterizes when existing heuristics help and why they fail, while showing that differential smoothing is universally superior. Extensive experiments with models from 1B to 7B parameters, across domains including CountDown and real-world mathematical reasoning, demonstrate consistent gains. Differential smoothing improves both Pass@1 and Pass@k, with up to 6.7% improvements on AIME24 dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。