通过追溯推理生成来源,揭示学生模型在测试时是否真正继承了教师模型的能力。
Where Did This Sentence Come From? Tracing Provenance in LLM Reasoning Distillation
- 构建跨模型溯源框架,比较教师、原始学生与蒸馏后学生的输出概率
- 发现蒸馏模型在测试中能生成源自教师的推理句子,且与性能提升相关
- 提出基于教师-学生差异的数据筛选方法,比传统启发式更可靠
推理蒸馏近年来受到广泛关注。通常用大教师模型生成推理路径,用于微调学生模型,使其在训练阶段模仿教师行为。然而,现有方法缺乏对学生模型能力来源的细致分析。尚不清楚学生模型在新测试场景下是否保持与教师一致的行为,或是否会退化回原始输出模式,引发对蒸馏模型泛化能力的担忧。为此,本文引入跨模型推理蒸馏溯源框架。对于蒸馏模型生成的每一项动作(如一句话),我们获取教师、原始学生及蒸馏模型在相同上下文下的预测概率。通过对比这些概率,将每项动作分类。系统性地解耦每个动作的来源后,实验表明:在测试场景中,蒸馏模型确实能生成源自教师的动作,且这些动作与观察到的性能表现相关联。基于此分析,进一步提出一种教师引导的数据选择方法。不同于依赖启发式的方法,该方法直接比较教师与学生在训练数据上的差异,提供原则性选择标准。我们在多个代表性教师模型和多样学生模型上验证了方法的有效性。结果凸显了溯源框架的实用性,并展现了其在推理蒸馏中的前景。
原文摘要 · Abstract (English)
Reasoning distillation has attracted increasing attention. It typically leverages a large teacher model to generate reasoning paths, which are then used to fine-tune a student model so that it mimics the teacher's behavior in training contexts. However, previous approaches have lacked a detailed analysis of the origins of the distilled model's capabilities. It remains unclear whether the student can maintain consistent behaviors with the teacher in novel test-time contexts, or whether it regresses to its original output patterns, raising concerns about the generalization of distillation models. To analyse this question, we introduce a cross-model Reasoning Distillation Provenance Tracing framework. For each action (e.g., a sentence) produced by the distilled model, we obtain the predictive probabilities assigned by the teacher, the original student, and the distilled model under the same context. By comparing these probabilities, we classify each action into different categories. By systematically disentangling the provenance of each action, we experimentally demonstrate that, in test-time contexts, the distilled model can indeed generate teacher-originated actions, which correlate with and plausibly explain observed performance on distilled model. Building on this analysis, we further propose a teacher-guided data selection method. Unlike prior approach that rely on heuristics, our method directly compares teacher-student divergences on the training data, providing a principled selection criterion. We validate the effectiveness of our approach across multiple representative teacher models and diverse student models. The results highlight the utility of our provenance-tracing framework and underscore its promise for reasoning distillation. We hope to share Reasoning Distillation Provenance Tracing and our insights into reasoning distillation with the community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。