失败推理轨迹可诊断问题是否可修复,无需重试或修改模型。
Failed Reasoning Traces Tell You What Is Fixable (But Not by Reading Them)

- 从失败轨迹的分布特征提取三类结构化信号,判断失败是否可通过重试修复。
- 在Steerable-Hard数据集上,故障修复率提升12.2%,准确率达84.3%±4.3%。
- 无需训练即可实现测试时路由,适用于多种后训练方法和模型家族。
当后训练语言模型在推理任务中失败时,常规做法是增加计算量进行更多尝试,而失败轨迹则被丢弃。我们指出,这些轨迹其实包含关键信息:部分失败源于随机采样误差,可通过更多尝试挽救;另一些则是结构性失败,无论增加多少计算都无法修复。本文提出,失败轨迹编码了可修复性结构——即在推理阶段,哪些干预措施能挽救特定失败的信号。通过三个源自可用干预结构的问题级轨迹特征,从失败轨迹的分布模式中提取该结构,而非其文本内容。这些特征将失败划分为稳定类型,刻画不同后训练方法的失败拓扑(准确率84.3%±4.3%,比多数类基线高20%),并支持一种无需训练的路由规则,在部署相关且仅靠重试无法解决的Steerable-Hard子集上,将救援成功率提升12.2%。该特征与路由规则在两个跨模型族探测器间具有良好迁移性。因此,这三个特征将原本被丢弃的失败轨迹转化为诊断工具,支持测试时路由与后训练分析,且无需训练或权重空间访问。
原文摘要 · Abstract (English)
When post-trained language models fail on reasoning problems, the common test-time-scaling response is to spend more compute on additional attempts, and the failed traces play no further role. We argue this discards a crucial signal; some failures come from unlucky sampling, where more rollouts help, while others are structural and resist resampling regardless of budget. We propose that failed traces encode recoverability structure: the inference-time signature of which test-time interventions can rescue a given failure. Three problem-level trajectory features, derived from the structure of available interventions, recover this structure from the distributional signature of failed rollouts, not their text. They cluster failures into stable regimes, characterize the failure topography of different post-training methods ($84.3{\pm}4.3\%$ accuracy, $+20\%$ over a majority-class baseline), and support a training-free routing rule that lifts rescue by $+12.2\%$ on the deployment-relevant Steerable-Hard subset (failures where retry is insufficient and a bounded intervention is reachable). The features and the routing rule transfer across two cross-family probes. The same three features thus convert failed traces from discarded data into a diagnostic object, supporting test-time routing and post-training analysis without training-time or weight-space access.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。