arXiv:2505.13792cs.CLcs.AI2025-05ACL被引 11

研究发现,大模型推理步骤既不准确也不易懂,需分开设计。

Interpretable Traces, Unexpected Outcomes: Investigating the Disconnect in Trace-Based Knowledge Distillation

  • 用规则分解问题,对比正确与错误推理步骤的影响
  • 28%的正确步骤仍导致错误答案,错误步骤也不总降低准确率
  • 越详细越难懂,简洁解释虽易读但效果差,需分开优化

近期基于推理的大型语言模型(LLMs)引入了思维链(CoT)轨迹——在给出最终答案前生成的中间推理步骤。这些轨迹如DeepSeek R1所示,用于指导推理并训练小型模型。一个常见但未被充分检验的假设是:这些轨迹在语义上正确且对终端用户可解释。尽管中间推理步骤被认为能提升准确性,我们质疑其实际有效性与可理解性。为隔离轨迹语义的影响,我们在问答任务中设计实验,采用基于规则的问题分解,构建微调数据集:每个问题配以可验证的正确或错误轨迹,始终提供正确最终答案。轨迹正确性通过逐个子步骤的准确性评估。为评估可解释性,我们在三种轨迹类型上微调LLM:R1轨迹、R1轨迹摘要和事后解释,并通过100名参与者的人类研究,在5分量表上评分。结果表明:(1) 轨迹正确性不能可靠预测最终答案正确性——正确轨迹仅在28%测试案例中带来正确解;错误轨迹也未一致降低准确率。(2) 在冗长的R1轨迹上微调取得最佳模型性能,但用户评分最低可解释性(3.39)和最高认知负荷(4.59);而更易懂的分解轨迹未能达到同等准确率。这些发现挑战了现有假设,提示研究人员与实践者应将模型监督目标与面向用户的轨迹设计分离。

原文摘要 · Abstract (English)

Recent advances in reasoning-focused Large Language Models (LLMs) have introduced Chain-of-Thought (CoT) traces - intermediate reasoning steps generated before a final answer. These traces, as in DeepSeek R1, guide inference and train smaller models. A common but under-examined assumption is that these traces are both semantically correct and interpretable to end-users. While intermediate reasoning steps are believed to improve accuracy, we question whether they are actually valid and understandable. To isolate the effect of trace semantics, we design experiments in Question Answering (QA) using rule-based problem decomposition, creating fine-tuning datasets where each problem is paired with either verifiably correct or incorrect traces, while always providing the correct final answer. Trace correctness is evaluated by checking the accuracy of every reasoning sub-step. To assess interpretability, we fine-tune LLMs on three additional trace types: R1 traces, R1 trace summaries, and post-hoc explanations, and conduct a human study with 100 participants rating each type on a Likert scale. We find: (1) Trace correctness does not reliably predict correct final answers - correct traces led to correct solutions in only 28% of test cases, while incorrect traces did not consistently degrade accuracy. (2) Fine-tuning on verbose R1 traces yielded the best model performance, but users rated them least interpretable (3.39 interpretability, 4.59 cognitive load on a 5-point scale), whereas more interpretable decomposed traces did not achieve comparable accuracy. Together, these findings challenge the assumption in question suggesting that researchers and practitioners should decouple model supervision objectives from end-user-facing trace design.

大模型可解释性知识蒸馏

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。