通过分阶段动态调整上下文长度，提升大模型推理训练效率
FastCuRL: Curriculum Reinforcement Learning with Stage-wise Context Scaling for Efficient Training R1-like Reasoning Models
- 设计分阶段上下文缩放的课程强化学习框架
- 在AIME 2024上达49.6%准确率，优于现有模型
- 仅用8卡单节点、50%训练步数即超越同类模型
提升大规模强化学习训练效率仍是核心挑战。本文研究上下文长度与训练数据复杂度对R1类推理模型（如DeepSeek-R1-Distill-Qwen-1.5B）强化学习扩展训练过程的影响。实验表明：(1) 通过控制上下文长度并根据输入提示长度筛选训练数据，可显著提升训练效率，获得更简洁的思维链；(2) 合理缩放上下文长度有助于缓解熵崩溃问题；(3) 精确选择上下文长度能实现高效的大语言模型训练与推理。基于此，我们提出FastCuRL，一种具有分阶段上下文缩放能力的课程强化学习框架。大量实验表明，FastCuRL-1.5B-V3在五个竞赛级基准上显著优于当前最优推理模型，在AIME 2024上达到49.6%准确率。此外，FastCuRL-1.5B-Preview在五个基准上超越DeepScaleR-1.5B-Preview，且仅使用单节点8张GPU和50%的训练步数。
原文摘要 · Abstract (English)
Improving training efficiency continues to be one of the primary challenges in large-scale Reinforcement Learning (RL). In this paper, we investigate how context length and the complexity of training data influence the RL scaling training process of R1-distilled reasoning models, e.g., DeepSeek-R1-Distill-Qwen-1.5B. Our experimental results reveal that: (1) simply controlling the context length and curating the training data based on the input prompt length can effectively improve the training efficiency of RL scaling, achieving better performance with more concise CoT; (2) properly scaling the context length helps mitigate entropy collapse; and (3) carefully choosing the context length facilitates achieving efficient LLM training and reasoning. Inspired by these insights, we propose FastCuRL, a curriculum RL framework with stage-wise context scaling to achieve efficient LLM training and reasoning. Extensive experimental results demonstrate that FastCuRL-1.5B-V3 significantly outperforms state-of-the-art reasoning models on five competition-level benchmarks and achieves 49.6% accuracy on AIME 2024. Furthermore, FastCuRL-1.5B-Preview surpasses DeepScaleR-1.5B-Preview on five benchmarks while only using a single node with 8 GPUs and a total of 50% of training steps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。