arXiv:2605.17026cs.LG2026-05被引 1

推理模型覆盖范围下降,源于训练数据中的多重合理路径决策点。

Why Do Reasoning Models Lose Coverage? The Role of Data and Forks in the Road

论文配图:Why Do Reasoning Models Lose Coverage? The Role of Data and Forks in the Road
图 1 · 摘自论文原文
  • 通过模拟决策点场景,发现训练数据中复杂分支路径导致覆盖度下降。
  • 决策点越密集,pass@k性能越差,且该现象在多种推理模式中均出现。
  • 改进数据设计与解码策略可部分缓解覆盖度损失,适合模型优化研究者参考。

大语言模型的进展催生了推理模型,其通过特定微调在复杂任务上表现优异。然而,尽管此类方法能提升pass@1准确率,先前研究发现其存在覆盖度收缩现象,即pass@k性能反而低于基础模型。本文探究基于SFT的后训练中推理收缩的原因,提出假设:该现象由微调数据特性驱动,尤其与模型遇到难以分辨的多路径推理场景(即“岔路”)有关。通过设计受控案例,模拟图分支中的不可辨识节点及不同推理模式下的决策点,追踪训练动态发现,覆盖度收缩与训练数据中决策点的出现频率高度相关。同时证明,通过针对性的数据合成设计和更具多样性鼓励的解码机制,可部分缓解该问题。研究揭示数据因素是推理模型收缩的关键驱动,并强调多样性感知设计的有效性。(实验数据与代码详见https://github.com/psunlpgroup/reasoning_forks)

原文摘要 · Abstract (English)

Recent progress in large language models has led to the emergence of reasoning models, which have shown strong performance on complex tasks through specialized fine-tuning procedures. While these methods reliably improve pass@1 accuracy, prior works have observed that they show a coverage shrinkage behavior, where pass@k degrades relative to the base model. In this paper, we investigate the cause of reasoning shrinkage under SFT-based post-training. We hypothesize that this behavior is driven by properties of the fine-tuning data, specifically related to decision points or "forks in the road" scenarios where model encounters indecipherable patterns with multiple valid reasoning paths. To test this hypothesis, we design controlled case studies that simulate such decision-point settings, spanning indecipherable nodes in graph branching, and reasoning modes. By tracking post-training dynamics in these settings, we find that the shrinkage phenomenon is tightly correlated with the prevalence of decision-point scenarios in the training data. We also demonstrate that this shrinkage behavior can be partially mitigated through targeted data synthesis design of decision-points and a more systematic diversity-encouraging decoding mechanism. Our findings identify data-centric factors as a key driver of shrinkage in reasoning models and highlight diversity-aware designs as an effective lever for controlling it. (Data and code for reproducing our experiments are available at https://github.com/psunlpgroup/reasoning_forks )

推理模型数据偏差覆盖度微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。