小模型靠无监督强化学习难提升推理能力，新方法通过分阶段训练解决此问题。
You Need Reasoning to Learn Reasoning: The Limitations of Label-Free RL in Weak Base Models
- 用渐进式难度课程学习和淘汰多数错误推理路径，提升自监督训练效果。
- 0.5B至7B参数的小模型在无标注强化学习下推理性能普遍低于基线。
- 适合资源受限场景中希望自建推理能力的开发者使用。
近期大语言模型进展表明，无监督强化学习（RL）可在无需外部监督的情况下增强推理能力。然而，这类方法在小型基础模型中的泛化性尚未被探索。本文系统研究了从0.5B到7B参数的不同规模模型上无标签强化学习的表现。实证分析显示：该方法高度依赖基础模型的预存推理能力，对较弱模型而言性能常低于基线。我们发现，小模型无法生成足够长或多样化的思维链以支持有效自省，且训练数据难度是成功的关键因素。为此，我们提出一种简单但有效的无标签强化学习方法，采用课程学习逐步引入更难问题，并在训练中屏蔽无多数共识的推理路径。同时设计数据筛选流程生成具有预设难度的样本。该方法在所有模型规模和推理能力下均实现稳定提升，为资源受限模型的推理能力自举提供了可行路径。代码已开源：https://github.com/BorealisAI/CuMa。
原文摘要 · Abstract (English)
Recent advances in large language models have demonstrated the promise of unsupervised reinforcement learning (RL) methods for enhancing reasoning capabilities without external supervision. However, the generalizability of these label-free RL approaches to smaller base models with limited reasoning capabilities remains unexplored. In this work, we systematically investigate the performance of label-free RL methods across different model sizes and reasoning strengths, from 0.5B to 7B parameters. Our empirical analysis reveals critical limitations: label-free RL is highly dependent on the base model's pre-existing reasoning capability, with performance often degrading below baseline levels for weaker models. We find that smaller models fail to generate sufficiently long or diverse chain-of-thought reasoning to enable effective self-reflection, and that training data difficulty plays a crucial role in determining success. To address these challenges, we propose a simple yet effective method for label-free RL that utilizes curriculum learning to progressively introduce harder problems during training and mask no-majority rollouts during training. Additionally, we introduce a data curation pipeline to generate samples with predefined difficulty. Our approach demonstrates consistent improvements across all model sizes and reasoning capabilities, providing a path toward more robust unsupervised RL that can bootstrap reasoning abilities in resource-constrained models. We make our code available at https://github.com/BorealisAI/CuMa
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。