通过分阶段强化学习提升通用推理模型性能,无需牺牲效率。
Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models
- 采用分域渐进式强化学习,按领域顺序训练,降低工程复杂度。
- 140亿参数模型在多个基准上超越教师模型,获信息学奥赛银牌。
- 适合需要高效、强推理能力的模型开发团队参考使用。
利用强化学习构建通用推理模型面临跨领域异质性挑战,包括推理响应长度和验证延迟差异大,导致训练基础设施复杂、速度慢,且难以设计训练课程与调参。本文提出分层域内强化学习(Cascade RL),用于开发Nemotron-Cascade,可在指令模式与深度思考模式间切换,性能与纯思考模式无差异。不同于传统混合多领域提示的方法,该方法按领域顺序分步进行强化学习,显著降低工程复杂度,并在广泛基准测试中达到顶尖表现。值得注意的是,预训练阶段的基于人类反馈的强化学习(RLHF)不仅优化偏好,更大幅提升模型推理能力;后续各域内强化学习阶段极少损害前期性能,甚至可能提升。我们140亿参数模型经强化学习后,在LiveCodeBench v5/v6/Pro上优于其SFT教师模型DeepSeek-R1-0528,且在2025年国际信息学奥林匹克竞赛(IOI)中取得银牌成绩。训练与数据方案已公开透明共享。
原文摘要 · Abstract (English)
Building general-purpose reasoning models with reinforcement learning (RL) entails substantial cross-domain heterogeneity, including large variation in inference-time response lengths and verification latency. Such variability complicates the RL infrastructure, slows training, and makes training curriculum (e.g., response length extension) and hyperparameter selection challenging. In this work, we propose cascaded domain-wise reinforcement learning (Cascade RL) to develop Nemotron-Cascade, capable of operating in both instruct and deep thinking modes, without any performance gap relative to a thinking-only counterpart. Departing from conventional approaches that blend heterogeneous prompts from different domains, Cascade RL orchestrates sequential, domain-wise RL, reducing engineering complexity and delivering state-of-the-art performance across a wide range of benchmarks. Notably, RLHF for alignment, when used as a pre-step, boosts the model's reasoning ability far beyond mere preference optimization, and subsequent domain-wise RLVR stages rarely degrade the benchmark performance attained in earlier domains and may even improve it (see an illustration in Figure 1). Our 14B model, after RL, outperforms its SFT teacher, DeepSeek-R1-0528, on LiveCodeBench v5/v6/Pro and achieves silver-medal performance in the 2025 International Olympiad in Informatics (IOI). We transparently share our training and data recipes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。