用强化学习让大模型回答更简洁，且不牺牲准确率
Concise Reasoning via Reinforcement Learning
- 通过两阶段强化学习，训练模型在简单问题上生成短答案
- 实验显示简洁回答与正确率正相关，长度减少40%以上
- 适合需要低延迟、低成本推理的场景
推理模型的一大缺陷是过度使用令牌，导致计算成本高、资源消耗大和响应延迟。我们发现这种冗长并非源于更深的推理，而是由于强化学习在错误答案上的损失最小化所致。训练中不可解问题占主导，导致模型系统性地倾向于生成更长输出。通过对PPO和GRPO的理论分析，我们证明即使折扣因子γ=1，错误答案也会引发冗长输出，将长度增加视为优化过程的副产品。进一步发现，无论是否具备推理能力，简洁性与正确性始终存在一致关联。基于此，我们提出一种两阶段强化学习方法：在小规模可解问题上训练一个简短的第二阶段，显著缩短输出长度，同时保持或提升准确率。最后，尽管GRPO与PPO有相似性质，但存在崩溃模式，限制其在简洁推理中的可靠性。实验充分验证了上述结论。
原文摘要 · Abstract (English)
A major drawback of reasoning models is their excessive token usage, inflating computational cost, resource demand, and latency. We show this verbosity stems not from deeper reasoning but from reinforcement learning loss minimization when models produce incorrect answers. With unsolvable problems dominating training, this effect compounds into a systematic tendency toward longer outputs. Through theoretical analysis of PPO and GRPO, we prove that incorrect answers inherently drive policies toward verbosity \textit{even when} $γ=1$, reframing response lengthening as an optimization artifact. We further uncover a consistent correlation between conciseness and correctness across reasoning and non-reasoning models. Building on these insights, we propose a two-phase RL procedure where a brief secondary stage, trained on a small set of solvable problems, significantly reduces response length while preserving or improving accuracy. Finally, we show that while GRPO shares properties with PPO, it exhibits collapse modes, limiting its reliability for concise reasoning. Our claims are supported by extensive experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。