让大模型根据正确答案反推推理链,会严重降低其逻辑推理能力。
Answer-Conditioned Chains of Thought Degrade Verifiable-Reasoning Distillation in Large Language Models

- 用正确答案引导模型生成推理链,导致推理过程倒退
- 在最难竞赛题上准确率下降最高达27点
- 适合训练逻辑推理模型的研究者警惕此陷阱
大语言模型推理能力的蒸馏标准方法是采样推理链,保留得出正确答案的样本进行微调。当采样失败时,常将正确答案提供给生成器,要求其写出能到达该答案的推理链。我们发现,这一步骤会以正确性过滤无法察觉的方式破坏训练数据。在控制变量实验中,仅改变推理链是否受答案条件约束(即是否看到正确答案并要求抵达),结果表明:仅使用自身答案条件链训练的强指令微调推理模型,其可验证推理准确率显著下降,且损失随题目难度增加,在最难题目上最高达27个百分点。分析显示,推理链会从答案倒推而非正向推导,早期出现最终答案是明显征兆。损害是数据本身的属性,可在无标签生成中预判,并在八种不同思维模型间传递,跨教师家族迁移。提示词消融实验证明,问题出在‘朝答案推理’指令本身,而非答案可见性。实际建议是:生成推理链应无视答案,因为任何正确性过滤都无法察觉此类损伤。
原文摘要 · Abstract (English)
A standard recipe for distilling the reasoning ability of large language models (LLMs) is to sample chains of thought from the model, keep those that reach the correct final answer, and fine-tune on the survivors. When sampling fails, a common fix shows the generator the gold answer and asks it to write a chain that reaches that answer. We show that this second step degrades the training data in a way that correctness filtering cannot catch. We run a controlled experiment that fixes the generator, the problem set, and the correctness filter, and varies only whether the chain is generated under answer-conditioning, the gold answer shown with a request to reach it. Training a strong instruction-tuned reasoning model on its own answer-conditioned chains sharply lowers its verifiable-reasoning accuracy. The loss grows with difficulty, reaching as much as about 27 points on the hardest competition problems. The mechanism is legible in the chains themselves, which rationalize backward from the shown answer instead of deriving it, with the early final-answer statement as the measurable symptom. The harm is a property of the data rather than the generator, read off unlabeled generations before any fine-tuning, ordering the penalty across eight thinking models from four families, and transferring across teacher families. A prompt ablation localizes it to the rationalize-toward instruction rather than the answer's bare visibility. The practical takeaway is to generate answer-blind, because no correctness filter can see this damage in the data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。