arXiv:2504.13837cs.AIcs.CL2025-04NeurIPS被引 1.0k

RL训练并未让大模型获得真正的新推理能力,反而被基模型上限束缚。

Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

  • 用大k值的通过率测试揭示:强化学习训练未催生新推理模式。
  • 大k值下基模型表现优于微调后模型,说明能力受限于基线。
  • 知识蒸馏可引入新推理路径,是突破现有瓶颈的有效方法。

强化学习结合可验证奖励(RLVR)近期在提升大语言模型(LLMs)的数学与编程推理能力方面取得显著成效。尽管传统强化学习能帮助智能体探索新策略,但人们普遍认为RLVR可使模型持续自我优化,从而获得超越基模型的新型推理能力。本研究系统评估了多种模型架构、强化学习算法及数学、编码和视觉推理基准下RLVR训练模型的推理边界,采用大k值的pass@k作为评估指标。令人惊讶的是,当前训练设置并未激发根本性的新推理模式。尽管在小k值(如k=1)时RLVR模型优于基模型,但在大k值下基模型表现更优。覆盖率与困惑度分析表明,观察到的推理能力始终源于并受限于基模型。以基模型为上限进行定量分析发现,六种主流RLVR算法表现相似且远未达到最优。相比之下,知识蒸馏能够从教师模型中引入全新推理模式,真正扩展模型推理能力。总体而言,当前RLVR方法尚未实现强化学习激发真正新推理能力的潜力,亟需改进的强化学习范式,如持续扩展和多轮智能体-环境交互,以释放这一潜能。

原文摘要 · Abstract (English)

Reinforcement Learning with Verifiable Rewards (RLVR) has recently demonstrated notable success in enhancing the reasoning performance of large language models (LLMs), particularly on mathematics and programming tasks. Similar to how traditional RL helps agents explore and learn new strategies, RLVR is believed to enable LLMs to continuously self-improve, thus acquiring novel reasoning abilities beyond those of the corresponding base models. In this study we critically examine the current state of RLVR by systematically probing the reasoning capability boundaries of RLVR-trained LLMs across various model families, RL algorithms, and math, coding, and visual reasoning benchmarks, using pass@k at large k values as the evaluation metric. Surprisingly, we find that the current training setup does not elicit fundamentally new reasoning patterns. While RLVR-trained models outperform their base models at small k (e.g., k = 1), the base models achieve a higher pass@k score when k is large. Coverage and perplexity analyses show that the observed reasoning abilities originate from and are bounded by the base model. Treating the base model as an upper bound, our quantitative analysis shows that six popular RLVR algorithms perform similarly and remain far from optimal in leveraging the potential of the base model. By contrast, we find that distillation can introduce new reasoning patterns from the teacher and genuinely expand the model's reasoning capabilities. Overall, our findings suggest that current RLVR methods have not yet realized the potential of RL to elicit truly novel reasoning abilities in LLMs. This highlights the need for improved RL paradigms, such as continual scaling and multi-turn agent-environment interaction, to unlock this potential.

强化学习推理能力大模型知识蒸馏

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。