对比不同训练方式如何让模型学会推理，发现强化学习主要教判断时机，监督微调则引入新推理机制。
Base Models Know How to Reason, Thinking Models Learn When
- 用稀疏自编码器提取推理行为模式，构建可解释的推理分类体系。
- 强化学习模型仅修复11%的推理差距，而监督微调能恢复76%，说明两者教学方式本质不同。
- 适合关注推理机制演化、高效构建思考型模型的研究者阅读。
本研究提出一种无监督方法，通过在推理轨迹的句级激活上训练小型稀疏自编码器，识别模型的推理行为并生成可解释的推理分类体系。在此基础上，我们引入构造性模型差异分析，旨在从可解释组件中重构基座模型到微调模型的差异：包括能触发推理行为的类别向量（推理机制）和决定何时启用机制的分类器（推理启发式）。在九组基座/思考模型对（四组强化学习训练，四组监督微调蒸馏，一组混合）中，两个独立发现一致：基座模型中的类别向量在纯强化学习推导的分类体系下损失更低；混合模型可恢复约76%的强化学习基座到思考模型差距，但仅恢复11%的监督微调差距。这表明强化学习主要教会模型如何协调已有机制（即启发式），而监督微调蒸馏则植入新的推理机制，为理解不同训练范式所传授内容提供了新视角，对高效推理模型开发具有重要启示。
原文摘要 · Abstract (English)
What do thinking language models learn during training that their base models lack? We first present an unsupervised method that discovers a model's reasoning behaviors by training small Sparse Autoencoders on sentence-level activations of reasoning traces, yielding interpretable reasoning taxonomies. Building on this, we introduce constructive model diffing, which aims to reconstruct the base-to-fine-tuned difference from interpretable components: reasoning mechanisms (category vectors that can induce a reasoning behavior in the base model) and reasoning heuristics (a classifier determining when a mechanism should fire). Across nine base/thinking pairs (four RL-trained, four SFT-distilled, one mixed), two independent findings agree: category vectors in the base model converge to far lower loss for taxonomies derived from purely RL-trained models, and hybrid models recover roughly 76% of the RL base-to-thinking gap but only 11% of the SFT gap. This indicates RL primarily teaches heuristics for orchestrating pre-existing base mechanisms, whereas SFT-distillation installs new ones, offering a new lens on what training paradigms teach, with implications for efficient reasoning-model development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。