TACLer通过渐进式训练提升大模型推理效率与准确率。
TACLer: Tailored Curriculum Reinforcement Learning for Efficient Reasoning
- 根据模型能力动态调整训练数据难度,分阶段学习
- 减少50%以上训练算力,推理节省42%以上token
- 适合需要高效高精度推理的复杂任务场景
大型语言模型在复杂推理任务中表现优异,尤其在采用长链思维(CoT)时。然而,生成长链思维通常需要大规模强化学习训练,且常导致冗余中间步骤的过度思考。为提高学习与推理效率,同时保持或提升性能,我们提出TACLer:一种面向模型特性的渐进式强化学习框架,依据模型在多阶段强化学习中的掌握程度逐步增加数据复杂度。TACLer包含两个核心组件:(i) 个性化渐进式学习,识别模型缺失的知识并分阶段补充;(ii) 混合思维/非思维推理范式,通过启用或禁用思维模式平衡准确率与效率。实验表明,TACLer在学习与推理上实现双重优势:(i) 训练计算成本降低超50%,推理令牌使用量相比基线模型减少超42%;(ii) 在四个含复杂问题的数学数据集上,准确率提升超9%,持续优于当前最优的无思维与有思维基线模型。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown remarkable performance on complex reasoning tasks, especially when equipped with long chain-of-thought (CoT) reasoning. However, eliciting long CoT typically requires large-scale reinforcement learning (RL) training, while often leading to overthinking with redundant intermediate steps. To improve learning and reasoning efficiency, while preserving or even enhancing performance, we propose TACLer, a model-tailored curriculum reinforcement learning framework that gradually increases the complexity of the data based on the model's proficiency in multi-stage RL training. TACLer features two core components: (i) tailored curriculum learning that determines what knowledge the model lacks and needs to learn in progressive stages; (ii) a hybrid Thinking/NoThinking reasoning paradigm that balances accuracy and efficiency by enabling or disabling the Thinking mode. Our experiments show that TACLer yields a twofold advantage in learning and reasoning: (i) it reduces computational cost, cutting training compute by over 50% compared to long thinking models and reducing inference token usage by over 42% relative to the base model; and (ii) it improves accuracy by over 9% on the base model, consistently outperforming state-of-the-art Nothinking and Thinking baselines across four math datasets with complex problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。