用简洁性奖励让大模型推理更高效，少算多对。
ConciseRL: Conciseness-Guided Reinforcement Learning for Efficient Reasoning Models
- 用大模型当裁判，动态评估推理过程的简洁性并给奖励。
- 数学题上减少31倍token，准确率提升7%，最难题多对7.5%。
- 自动调节推理长度，适合追求高效精准的AI系统开发者。
大语言模型通过分步推理完成复杂任务，但推理过程常冗长，导致计算浪费、可读性差和幻觉问题。为此，我们提出一种无需超参数调优的简洁性评分，作为强化学习中的奖励信号，引导模型生成正确且简洁的推理链。该评分由大模型充当裁判，实现超越单纯词元长度的上下文感知反馈。在MATH数据集上，简单问题可减少31倍的词元使用量，准确率提升7%；最难题目上，准确率高出完整推理7.5%，同时节省3.6倍词元。在TheoremQA上，准确率提升2.2%，词元消耗降低12.5倍。消融实验表明，方法能根据题目难度自适应调整推理长度，且更强的裁判模型带来显著收益。代码、模型权重与数据集已开源（https://github.com/RazvanDu/ConciseRL）。
原文摘要 · Abstract (English)
Large language models excel at complex tasks by breaking down problems into structured reasoning steps. However, reasoning traces often extend beyond reaching a correct answer, causing wasted computation, reduced readability, and hallucinations. To address this, we introduce a novel hyperparameter-free conciseness score used as a reward signal within a reinforcement learning framework to guide models toward generating correct and concise reasoning traces. This score is evaluated by a large language model acting as a judge, enabling dynamic, context-aware feedback beyond simple token length. Our method achieves state-of-the-art efficiency-accuracy trade-offs on the MATH dataset, reducing token usage by up to 31x on simple problems while improving accuracy by 7%, and on the hardest problems, it outperforms full reasoning by +7.5% accuracy with up to 3.6x fewer tokens. On TheoremQA, our method improves accuracy by +2.2% using 12.5x fewer tokens. We also conduct ablation studies on the judge model, reward composition, and problem difficulty, showing that our method dynamically adapts reasoning length based on problem difficulty and benefits significantly from stronger judges. The code, model weights, and datasets are open-sourced at https://github.com/RazvanDu/ConciseRL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。