arXiv:2603.11193cs.CL2026-03被引 1

按难度拆分训练数据,让大模型先学基础再攻坚难题,提升通用推理能力。

DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoning

  • 根据大模型评分将题目分为难易两类,分别用于监督微调和强化学习
  • 在多个科学与数学基准上,性能超越纯微调、纯强化学习及随机划分方案
  • 适合需要复杂推理的通用领域模型优化,尤其适用于数学与科学任务

强化学习结合可验证奖励(RLVR)已成为激发大语言模型推理能力的有效范式,尤其在数学和编程领域表现突出。尽管近期研究已将其扩展至更广泛的通用科学(STEM)领域,但监督微调(SFT)与强化学习(RL)在这些场景下的相互作用仍缺乏系统探索。本文通过受控实验发现:在通用STEM领域,直接对基础模型应用强化学习样本效率极低,且始终不如中等质量响应的监督微调表现;然而,顺序执行SFT后接RL可进一步提升性能,表明两阶段具有互补性,且数据分配方式至关重要。为此,我们提出DeReason,一种基于难度的数据解耦策略。该方法利用大模型评分估计推理强度,将训练数据划分为高推理强度与低推理强度子集:前者用于强化学习以培养复杂推理能力,后者分配给监督微调以建立领域知识基础。实验表明,这种有原则的解耦策略优于随机划分,显著提升序列化SFT-then-RL训练效果。在通用STEM与数学基准上的广泛测试验证了其优越性,为通用推理的后训练提供了高效且通用的解决方案。

原文摘要 · Abstract (English)

Reinforcement learning with Verifiable Rewards (RLVR) has emerged as a powerful paradigm for eliciting reasoning capabilities in large language models, particularly in mathematics and coding. While recent efforts have extended this paradigm to broader general scientific (STEM) domains, the complex interplay between supervised fine-tuning (SFT) and RL in these contexts remains underexplored. In this paper, we conduct controlled experiments revealing a critical challenge: for general STEM domains, RL applied directly to base models is highly sample-inefficient and is consistently surpassed by supervised fine-tuning (SFT) on moderate-quality responses. Yet sequential SFT followed by RL can further improve performance, suggesting that the two stages play complementary roles, and that how training data is allocated between them matters. Therefore, we propose DeReason, a difficulty-based data decoupling strategy for general reasoning. DeReason partitions training data by reasoning intensity estimated via LLM-based scoring into reasoning-intensive and non-reasoning-intensive subsets. It allocates broad-coverage, non-reasoning-intensive problems to SFT to establish foundational domain knowledge, and reserves a focused subset of difficult problems for RL to cultivate complex reasoning. We demonstrate that this principled decoupling yields better performance than randomly splitting the data for sequential SFT and RL. Extensive experiments on general STEM and mathematical benchmarks demonstrate that our decoupled curriculum training significantly outperforms SFT-only, RL-only, and random-split baselines. Our work provides a systematic study of the interplay between SFT and RL for general reasoning, offering a highly effective and generalized post-training recipe.

大模型训练推理增强强化学习数据解耦

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。