用模型自适应选题训练,显著降低推理模型的标注与算力成本。
Learning to Reason with Curriculum I: Provable Benefits of Autocurriculum
- 模型自主挑选难例进行训练,动态调整学习重点。
- 监督微调所需推理示范量呈指数级减少。
- 强化学习阶段可摆脱对高质量参考模型的依赖。
思维链推理通过让语言模型生成思考过程来提升能力,但训练成本高昂,需大量人工或合成的推理轨迹,并通过强化学习进一步优化。本文证明:自适应课程(autocurriculum)——模型根据自身表现自动选择训练问题——能显著优于传统训练方法。在监督微调中,该方法仅需指数级更少的推理示范,即可达到相同效果;在强化学习中,计算成本不再依赖参考模型质量,其影响仅限于初始阶段。这些优势源于自适应数据选择机制,结合了提升算法和反例学习的经典思想,且不依赖提示分布或难度假设。
原文摘要 · Abstract (English)
Chain-of-thought reasoning, where language models expend additional computation by producing thinking tokens prior to final responses, has driven significant advances in model capabilities. However, training these reasoning models is extremely costly in terms of both data and compute, as it involves collecting long traces of reasoning behavior from humans or synthetic generators and further post-training the model via reinforcement learning. Are these costs fundamental, or can they be reduced through better algorithmic design? We show that autocurriculum, where the model uses its own performance to decide which problems to focus training on, provably improves upon standard training recipes for both supervised fine-tuning (SFT) and reinforcement learning (RL). For SFT, we show that autocurriculum requires exponentially fewer reasoning demonstrations than non-adaptive fine-tuning, by focusing teacher supervision on prompts where the current model struggles. For RL fine-tuning, autocurriculum decouples the computational cost from the quality of the reference model, reducing the latter to a burn-in cost that is nearly independent of the target accuracy. These improvements arise purely from adaptive data selection, drawing on classical techniques from boosting and learning from counterexamples, and requiring no assumption on the distribution or difficulty of prompts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。