微调导致推理过程消失,模型答对但没了逻辑链条。
Reasoning-Trace Collapse: Evaluating the Loss of Explicit Reasoning During Fine-Tuning

- 用结构化评估框架区分答案正确性和推理有效性。
- 标准微调使有效推理链下降超70%,但答案准确率仍高。
- 仅通过损失掩码即可缓解推理崩溃,无需教师推理数据。
显式推理模型在给出最终答案前会生成中间推理步骤,但下游微调常使用不含推理痕迹的普通指令-回答数据。我们发现这种不匹配会导致推理痕迹崩溃:微调后的模型仍能给出合理答案,却失去了使其成为推理模型的关键结构化推理过程。为此,我们提出一种结构化评估框架,可分别衡量答案正确性与推理痕迹的有效性,包括有效、空、缺失和截断等状态,并评估基于有效推理的任务性能。通过对四个开源推理模型的研究发现,标准监督微调会迅速抑制有效推理痕迹,而仅依赖答案正确率的指标会严重掩盖这一问题:在多个场景中,基于有效推理的性能仍保持高位,但有效推理率急剧下降。此外,我们证明简单的损失掩码策略可显著缓解崩溃现象,且无需依赖教师生成的推理痕迹。结果表明,评估微调后的推理模型时,应同时报告结构化推理可靠性指标,尤其当训练数据不含显式推理痕迹时。
原文摘要 · Abstract (English)
Explicit reasoning models are trained to produce intermediate reasoning traces before final answers, but downstream fine-tuning is often performed on ordinary instruction-response data that contains no such traces. We show that this mismatch can induce reasoning-trace collapse: a fine-tuned model continues to produce plausible final answers while losing the structurally valid explicit reasoning traces that made it a reasoning model in the first place. We introduce a structural evaluation framework that separates answer correctness from reasoning-trace validity, measuring valid, empty, missing, and truncated reasoning alongside reasoning-conditioned task performance. Using this framework, we study four open-weight reasoning models and find that standard supervised fine-tuning can rapidly suppress valid reasoning traces, and that answer-only metrics can substantially obscure this failure: in several settings, performance conditional on valid reasoning remains high while the rate of valid reasoning falls sharply. We further show that simple loss-masking strategies can substantially mitigate collapse without requiring teacher-generated reasoning traces. These results suggest that evaluations of fine-tuned reasoning models should report structural reasoning reliability metrics in addition to final-answer performance, especially when adaptation data does not contain explicit reasoning traces.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。