arXiv:2508.08940cs.CLcs.AI2025-08被引 23

通过渐进式压缩训练,让大模型学会用更少的步骤高效推理。

Train Long, Think Short: Curriculum Learning for Efficient Reasoning

  • 从宽预算开始逐步收紧,模仿学习中从探索到精炼的过程。
  • 在GSM8K等数据集上,相同最终预算下准确率更高,节省超30%计算量。
  • 适合想提升模型推理效率的研究者与工程师。

近期研究通过控制生成长度来降低大语言模型的推理成本,但现有方法采用固定长度预算,未能利用学习过程中从探索到压缩的自然演进。本文提出基于组相对策略优化(GRPO)的课程学习策略:初始设置宽松的词元预算,随训练逐步收紧,促使模型先发现有效解题路径,再提炼为更简洁的推理过程。通过融合验证器反馈、长度效率和结构标签的奖励函数,平衡任务正确性、效率与格式规范。在GSM8K、MATH500、SVAMP、College Math和GSM+上的实验表明,该方法在相同最终预算下优于固定预算基线,显著提升准确率与词元效率。消融实验显示,渐进约束是训练高效推理模型的强大归纳偏置。代码与模型已开源。

原文摘要 · Abstract (English)

Recent work on enhancing the reasoning abilities of large language models (LLMs) has introduced explicit length control as a means of constraining computational cost while preserving accuracy. However, existing approaches rely on fixed-length training budgets, which do not take advantage of the natural progression from exploration to compression during learning. In this work, we propose a curriculum learning strategy for length-controlled reasoning using Group Relative Policy Optimization (GRPO). Our method starts with generous token budgets and gradually tightens them over training, encouraging models to first discover effective solution strategies and then distill them into more concise reasoning traces. We augment GRPO with a reward function that balances three signals: task correctness (via verifier feedback), length efficiency, and formatting adherence (via structural tags). Experiments on GSM8K, MATH500, SVAMP, College Math, and GSM+ demonstrate that curriculum-based training consistently outperforms fixed-budget baselines at the same final budget, achieving higher accuracy and significantly improved token efficiency. We further ablate the impact of reward weighting and decay schedule design, showing that progressive constraint serves as a powerful inductive bias for training efficient reasoning models. Our code and checkpoints are released at: https://github.com/hammoudhasan/curriculum_grpo.

课程学习推理优化大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。