发现强化学习训练的语言模型存在最佳推理长度，过长反而降低准确率。
On the Optimal Reasoning Length for RL-Trained Language Models
- 在数学和代码生成任务中测试多种长度控制策略，发现准确率随输出长度先升后降。
- 模型在中间长度时准确率最高，但模式准确率仍随长度持续提升。
- 揭示了非单调关系源于推理路径向更正确中心收敛，适合优化推理效率的研究者参考。
强化学习显著提升大语言模型的推理能力，但也导致思维链输出变长并增加计算成本。尽管已有长度控制方法，其引发的长度-准确率关系仍不明确。我们在受控环境中对多个基础模型使用多种长度控制方法训练策略，发现在数学推理与代码生成任务中，准确率随输出长度呈现非单调变化，在中间长度时达到峰值。然而，模式准确率在样本准确率趋于平稳或下降的设置下仍持续提升，表明非单调关系是由推理路径向越来越正确的中心聚集所驱动。
原文摘要 · Abstract (English)
Reinforcement learning substantially improves reasoning in large language models, but it also tends to lengthen chain-of-thought outputs and increase computational cost. Although length-control methods have been proposed, the length-accuracy relationship they induce remains unclear. We train policies with several length-control methods on multiple base models in a controlled setup and find that, across both mathematical reasoning and code generation, accuracy is non-monotonic in output length, peaking at an intermediate value. Mode accuracy, however, continues to improve with length even in settings where sample accuracy plateaus or declines, indicating that the non-monotonic length-accuracy relationship is driven by dispersion around an increasingly correct center.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。