课程学习让大模型推理更高效，理论证明可大幅降低训练样本需求。
Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Training
- 将推理过程建模为状态驱动的树状结构，设计渐进式任务提升策略。
- 课程学习使样本复杂度从指数级降至多项式级，显著提升训练效率。
- 适用于需要强逻辑推理能力的研究者，尤其关注模型训练优化者。
大语言模型后训练阶段的课程学习方法在提升推理性能方面已被实证优于非课程方法，但其有效性的理论机制尚不清晰。为此，本文构建了一个抽象理论框架，揭示了课程学习在样本复杂度上实现指数级提升的充分条件。我们把基础模型的思维链生成建模为状态条件下的自回归推理树，并将课程子任务形式化为两种策略：深度递增型（逐步扩展推理深度）与提示递减型（逐步减少部分提示）。分析表明，采用这两种课程策略的强化学习微调可在多项式样本复杂度下达到高精度，而非课程方法则面临指数级复杂度瓶颈。我们进一步建立了测试时缩放的类似理论保证。实验模拟验证了理论结论。代码已开源于 https://github.com/DakeBU/Curriculum-Post-training。
原文摘要 · Abstract (English)
Recent curriculum techniques in the post-training stage of LLMs have been empirically observed to outperform non-curriculum approaches in improving reasoning performance, yet a principled understanding of their effectiveness and limitations remains incomplete. To bridge this gap, we develop an abstract theoretical framework and identify sufficient conditions under which curriculum post-training yields exponential improvements in sample complexity. To substantiate this framework, we model the base model's Chain-of-Thought generation as a state-conditioned autoregressive reasoning tree, and formalize curriculum subtasks as either depth-increasing curricula that progressively extend reasoning horizons or hint-decreasing curricula that gradually remove partial hints. Our analysis shows that reinforcement learning finetuning with both curriculum strategies achieves high accuracy with polynomial sample complexity, whereas non-curriculum counterpart encounters an exponential complexity bottleneck. We further establish analogous guarantees for test-time scaling. Empirical simulations support our theoretical findings. Code is available at https://github.com/DakeBU/Curriculum-Post-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。