让大模型自动调整题目难度,边学边优化训练路径。
CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning
- 根据模型表现动态分层题目,自动简化难题、丰富中等题。
- 在数学和跨领域推理上比现有方法高7.75至10.21分。
- 无需人工改题,自动生成训练数据,适合强化学习研究者。
在线强化学习结合可验证奖励(RLVR)已成为提升大语言模型推理能力的有效范式,但多数方法对固定问题集进行优化,导致求解过程浪费在已解决或过于困难的问题上。我们提出【CLPO(课程学习融合策略优化)】,一个自演化课程框架,利用在线回放准确率识别已解决、中等难度和困难问题,并根据模型当前能力重构任务。困难问题被简化以可学习,中等难度问题则多样化以提供有效训练变体。使学习课程随策略能力变化而动态演进,而非保持静态。重构过程不作为静态数据增强,而是通过重写后问题的下游准确率增益分配信用,无需额外人工标注,仅需原始可验证答案。在数学推理与跨域通用推理基准测试中,CLPO在Qwen3-8B上分别显著优于GRPO和DAPO,平均提升10.21和7.75分。消融实验表明,重构方式与重写损失均对性能提升有贡献,证明CLPO为激发更强推理能力提供了可扩展、鲁棒的自演化路径。
原文摘要 · Abstract (English)
Online reinforcement learning with verifiable rewards (RLVR) has become an effective paradigm for improving the reasoning abilities of large language models, but most methods still optimize reasoning trajectories over the static problem set, wasting rollout budget on solved or overly difficult problems. We propose \textbf{CLPO (Curriculum Learning meets Policy Optimization)}, a self-evolving curriculum framework that uses on-policy rollout accuracy to identify solved, medium-difficulty, and hard problems, then restructures selected tasks according to the model's current capability. Hard problems are simplified to become learnable, while medium-difficulty problems are diversified to provide useful training variation. This allows the learning curriculum to co-evolve with the policy rather than remaining fixed as the model's capability boundary shifts. Rather than treating these rewrites as static data augmentation, CLPO optimizes restructuring trajectories with credit assigned by the downstream accuracy gain of the rewritten problem, requiring no additional human annotations beyond the original verifiable answers. Experiments across mathematical reasoning and out-of-domain general reasoning benchmarks show that CLPO substantially outperforms GRPO and DAPO on Qwen3-8B by 10.21 and 7.75 average points, respectively. Ablation studies on math and code domains further show that both the restructuring mode and the rewriting loss contribute to the final gains, demonstrating that CLPO provides a scalable and robust pathway for eliciting stronger reasoning capabilities through a self-evolving curriculum.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。