不同推理路径影响模型泛化，收敛性路径更优。
On the Role of Reasoning Patterns in the Generalization Discrepancy of Long Chain-of-Thought Supervised Fine-Tuning
- 对比两种模型生成的推理轨迹，发现路径模式差异影响泛化性能。
- 基于发散路径训练的模型泛化能力差，过滤冗余分支后性能提升5.1%。
- 适合研究推理过程、模型训练数据质量的从业者参考。
长链推理（CoT）监督微调已成为构建大型推理模型的关键阶段。然而，不同来源的CoT轨迹如何影响模型泛化性能仍不清楚。本文通过控制问题集一致，比较由两个竞争模型 exttt{DeepSeek-R1-0528} 与 exttt{gpt-oss-120b} 生成的验证版CoT轨迹。尽管二者表现相近，却揭示出显著矛盾： exttt{DeepSeek-R1-0528} 数据训练损失更低，但推理基准测试中泛化性能明显更差。分析显示， exttt{gpt-oss-120b} 的轨迹高度收敛且具演绎性，而 exttt{DeepSeek-R1-0528} 倾向于发散和多分支探索。由此导致以 exttt{DeepSeek-R1} 数据训练的模型继承低效探索行为，易陷入冗余分支。为此提出仅保留少分支轨迹进行微调的简单方法，实验表明在 AIME25 上性能提升最高达 5.1%，在 BeyondAIME 上提升 5.5%，五项基准平均提升 3.6%。
原文摘要 · Abstract (English)
Supervised Fine-Tuning (SFT) on long Chain-of-Thought (CoT) trajectories has become a pivotal phase in building large reasoning models. However, how CoT trajectories from different sources influence the generalization performance of models remains an open question. In this paper, we conduct a comparative study using two sources of verified CoT trajectories generated by two competing models, \texttt{DeepSeek-R1-0528} and \texttt{gpt-oss-120b}, with their problem sets controlled to be identical. Despite their comparable performance, we uncover a striking paradox: lower training loss does not translate to better generalization. SFT on \texttt{DeepSeek-R1-0528} data achieves remarkably lower training loss, yet exhibits significantly worse generalization performance on reasoning benchmarks compared to those trained on \texttt{gpt-oss-120b}. To understand this paradox, we perform a multi-faceted analysis probing token-level SFT loss and step-level reasoning behaviors. Our analysis reveals a difference in reasoning patterns. \texttt{gpt-oss-120b} exhibits highly convergent and deductive trajectories, whereas \texttt{DeepSeek-R1-0528} favors a divergent and branch-heavy exploration pattern. Consequently, models trained with \texttt{DeepSeek-R1} data inherit inefficient exploration behaviors, often getting trapped in redundant exploratory branches that hinder them from reaching correct solutions. Building upon this insight, we propose a simple yet effective remedy of filtering out frequently branching trajectories to improve the generalization of SFT. Experiments show that training on selected \texttt{DeepSeek-R1-0528} subsets surprisingly improves reasoning performance by up to 5.1% on AIME25, 5.5% on BeyondAIME, and on average 3.6% on five benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。