自适应课程强化学习，解决大模型推理训练中的难度错配与遗忘问题。
AdaCuRL: Adaptive Curriculum Reinforcement Learning with Invalid Sample Mitigation and Historical Revisiting
- 基于粗到精难度估计动态调整训练课程，匹配模型能力。
- 在多个推理基准上显著提升大模型与多模态大模型性能。
- 支持数据回访与策略稳定机制，适合复杂推理任务优化。
强化学习在提升大语言模型推理能力方面展现出巨大潜力。然而，直接在混合难度样本上训练时，现有方法常面临梯度饥饿和策略退化问题。尽管已有研究采用思维链（CoT）数据，但高质量标注的构建仍耗时费力。另一种课程学习策略虽被探索，却普遍存在难度不匹配、依赖人工设计课程及灾难性遗忘等挑战。为此，我们提出AdaCuRL——一种自适应课程强化学习框架，结合粗到精难度估计与自适应课程调度，动态对齐数据难度与模型能力，并引入数据回访机制缓解灾难性遗忘。此外，通过自适应参考与稀疏KL策略防止策略退化。在多个推理基准上的大量实验表明，AdaCuRL在大语言模型与多模态大语言模型上均实现显著性能提升。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has demonstrated considerable potential for enhancing reasoning in large language models (LLMs). However, existing methods suffer from Gradient Starvation and Policy Degradation when training directly on samples with mixed difficulty. To mitigate this, prior approaches leverage Chain-of-Thought (CoT) data, but the construction of high-quality CoT annotations remains labor-intensive. Alternatively, curriculum learning strategies have been explored but frequently encounter challenges, such as difficulty mismatch, reliance on manual curriculum design, and catastrophic forgetting. To address these issues, we propose AdaCuRL, a Adaptive Curriculum Reinforcement Learning framework that integrates coarse-to-fine difficulty estimation with adaptive curriculum scheduling. This approach dynamically aligns data difficulty with model capability and incorporates a data revisitation mechanism to mitigate catastrophic forgetting. Furthermore, AdaCuRL employs adaptive reference and sparse KL strategies to prevent Policy Degradation. Extensive experiments across diverse reasoning benchmarks demonstrate that AdaCuRL consistently achieves significant performance improvements on both LLMs and MLLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。