RL提升大模型数学推理,不靠执行外部计划,而靠内建策略与知识整合。
Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning
- 用三维度框架拆解RL对推理的影响:计划执行、知识整合、子问题链处理。
- 给模型提供人工计划反而降低性能,但RL模型更抗干扰,鲁棒性显著提升。
- 提出新训练流程SparkleRL-PSS,用部分提示重用难题,无需额外数据生成。
强化学习(RL)已成为提升语言模型在复杂推理任务中表现的主流范式。尽管基于RL的训练方法如GRPO展现出显著的实证提升,但对其为何以及如何增强性能的细粒度理解仍不足。为此,我们提出SPARKLE——一个细粒度分析框架,从三个关键维度剖析RL的影响:(1) 计划遵循与执行,(2) 知识整合,(3) 子问题链处理。利用该框架,我们获得超越准确率的洞察。例如,提供显式的人工步骤计划反而会显著降低最困难基准上的表现,而经RL调优的模型表现出更强的鲁棒性,性能下降远小于基础模型或SFT模型。这表明RL并非主要提升对外部计划的执行能力,而是增强模型制定并遵循更适合自身推理过程的内部策略。相反,我们发现RL能有效提升模型将外部知识融入推理过程的能力,在多种任务中均取得一致增益。最后,我们研究了难以产生有效强化信号的复杂问题——那些导致混合质量推理轨迹的问题——是否仍可用于训练。为此,我们提出SparkleRL-PSS,一种多阶段RL流程,通过部分步骤引导重用困难问题,有效指导探索,且无需额外数据生成。我们的发现为理解RL如何塑造模型行为提供了原则性基础,并为构建更自适应、数据高效且可解释的推理型RL流水线提供了实践洞见。代码、数据与检查点已公开:https://sparkle-reasoning.github.io/。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has become the dominant paradigm for improving the performance of language models on complex reasoning tasks. Despite the substantial empirical gains demonstrated by RL-based training methods like GRPO, a granular understanding of why and how RL enhances performance is still lacking. To bridge this gap, we introduce SPARKLE, a fine-grained analytic framework to dissect the effects of RL across three key dimensions: (1) plan following and execution, (2) knowledge integration, and (3) chain of subproblems. Using this framework, we gain insights beyond mere accuracy. For instance, providing models with explicit human-crafted, step-by-step plans can surprisingly degrade performance on the most challenging benchmarks, yet RL-tuned models exhibit greater robustness, experiencing markedly smaller performance drops than base or SFT models. This suggests that RL may not primarily enhance the execution of external plans but rather empower models to formulate and follow internal strategies better suited to their reasoning processes. Conversely, we observe that RL enhances models' ability to integrate provided knowledge into their reasoning process, yielding consistent gains across diverse tasks. Finally, we study whether difficult problems -- those yielding no RL signals and mixed-quality reasoning traces -- can still be effectively used for training. We introduce SparkleRL-PSS, a multi-stage RL pipeline that reuses hard problems with partial step scaffolding, guiding exploration effectively without additional data generation. Together, our findings provide a principled foundation for understanding how RL shapes model behavior, offering practical insights for building more adaptive, data-efficient, and interpretable RL pipelines for reasoning tasks. Our code, data, and checkpoints are available at: https://sparkle-reasoning.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。