arXiv:2509.21128cs.AI2025-09中稿 · ICLR被引 33

对比强化学习与监督微调如何影响大模型的推理路径,揭示两者互补机制。

RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs

  • 通过轨迹与步骤两级分析,量化推理路径变化
  • 强化学习使错误路径压缩,正确路径扩展至约3倍
  • 适合研究推理机制、优化训练策略的AI开发者

大型语言模型通常通过可验证奖励的强化学习(RLVR)和基于推理轨迹的监督微调(SFT)来提升推理能力,但两种方法如何塑造推理过程仍不明确。本文提出新分析框架,量化不同训练方式下1.5B、7B、14B参数模型在数学领域的推理路径变化。从轨迹级(完整推理输出)与步骤级(推理图节点对应单步)两层分析发现:聚类显示,强化学习压缩错误路径,而监督微调将正确路径扩展至约3倍;步骤级分析表明,强化学习使节点访问频率、度及介数中心性衰减率陡升约2.5倍,而监督微调将其降至约1/3,说明强化学习聚焦少数关键步骤,监督微调则均匀分布功能。多角度拓扑评估揭示了二者共性与差异。本研究解释了当前两阶段训练(先SFT后RL)成功的原理,并为数据构建与高效学习提供实践启示。

原文摘要 · Abstract (English)

Large language models (LLMs) are typically trained by reinforcement learning (RL) with verifiable rewards (RLVR) and supervised fine-tuning (SFT) on reasoning traces to improve their reasoning abilities. However, how these methods shape reasoning capabilities remains largely elusive. Going beyond an accuracy-based investigation of how these two components sculpt the reasoning process, this paper introduces a novel analysis framework that quantifies reasoning paths and captures their qualitative changes under each training process (with models of 1.5B, 7B, and 14B parameters on mathematical domains). Specifically, we investigate the reasoning process at two levels of granularity: the trajectory-level, which examines complete reasoning outputs, and the step-level, which analyzes reasoning graphs whose nodes correspond to individual reasoning steps. Notably, clustering of unique reasoning trajectories shows complementary effects: RL compresses incorrect trajectories, whereas SFT expands correct ones. Step-level analysis reveals that RL steepens (about 2.5 times), while SFT flattens (reduced to about one-third), the decay rates of node visitation frequency, degree, and betweenness centrality distributions in the reasoning graph. This indicates that RL concentrates reasoning functionality into a small subset of steps, while SFT homogenizes it across many steps. Furthermore, by evaluating the reasoning graph topologies from multiple perspectives, we delineate the shared and distinct characteristics of RL and SFT. Our work presents a novel reasoning path perspective that explains why the current best practice of two-stage training, with SFT followed by RL, is successful, and offers practical implications for data construction and more efficient learning approaches.

推理机制强化学习模型分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。