让大模型从易到难逐步训练，提升推理能力
Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning
- 按由易到难顺序调度任务，分阶段培养推理能力
- 小模型（1.5B~3B）在数学编码任务上性能显著提升
- 理论证明收敛性，且总样本需求少于直接训练
我们旨在通过强化学习（RL）提升语言模型的推理能力。尽管近期如DeepSeek-R1等后训练模型已在数学与编程任务中展现推理能力，但纯强化学习在处理本质困难任务时效果有限。受课程学习启发，我们提出从易到难（E2H）的任务调度策略，使大模型逐步建立推理技能。实证发现，初始阶段使用简单任务至关重要，但需适时淡出以防止过拟合。理论上，我们在近似策略迭代框架下建立了E2H Reasoner的收敛性保证，并推导出有限样本复杂度界，表明当任务合理分解与条件化时，分阶段学习所需总样本数少于直接学习。跨多个领域的实验表明，该方法显著提升了小规模语言模型（1.5B至3B）的推理能力，而这些模型在使用传统强化学习时表现不佳。代码已公开于https://github.com/divelab/E2H-Reasoning。
原文摘要 · Abstract (English)
We aim to improve the reasoning capabilities of language models via reinforcement learning (RL). Recent RL post-trained models like DeepSeek-R1 have demonstrated reasoning abilities on mathematical and coding tasks. However, prior studies suggest that using RL alone to improve reasoning on inherently difficult tasks is less effective. Here, we draw inspiration from curriculum learning and propose to schedule tasks from easy to hard (E2H), allowing LLMs to build reasoning skills gradually. Our method is termed E2H Reasoner. Empirically, we observe that, although easy tasks are important initially, fading them out through appropriate scheduling is essential in preventing overfitting. Theoretically, we establish convergence guarantees for E2H Reasoner within an approximate policy iteration framework. We derive finite-sample complexity bounds and show that when tasks are appropriately decomposed and conditioned, learning through curriculum stages requires fewer total samples than direct learning. Experiments across multiple domains show that E2H Reasoner significantly improves the reasoning ability of small LLMs (1.5B to 3B), which otherwise struggle when trained with vanilla RL alone, highlighting the effectiveness of our method. Our code can be found on https://github.com/divelab/E2H-Reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。