arXiv:2601.04263cs.LGcs.AI2026-01被引 2

让小模型学会大模型的推理逻辑,提升预测一致性与可解释性。

Learning to Reason: Temporal Saliency Distillation for Interpretable Knowledge Transfer

  • 通过时间显著性蒸馏,将教师模型的推理依据传递给学生模型。
  • 学生模型预测分布更贴近教师,显著减少分布差异。
  • 无需额外参数,适合对可解释性要求高的时序分析场景。

知识蒸馏在模型压缩中有效,通常将大型教师网络的知识转移到小型学生网络。现有时序数据的知识蒸馏主要沿用计算机视觉中的logit与特征对齐方法,未显式考虑时间序列特性,存在两大缺陷:其一,由于logit与特征不可解释,知识转移如何帮助学生学习不明确;其二,仅复制教师的预测准确率,知识传递有限,导致学生模型的预测分布与教师显著不同,难以安全替代。本文提出通过扩展传统logit蒸馏,不仅传递正确预测,更传递教师的合理推理过程。具体地,从教师logit中提取时间显著性,捕捉每个输入时间步对预测的重要性。通过时间显著性蒸馏训练学生模型,使其基于与教师相同的输入特征进行预测。该方法无需额外参数或特定架构假设。实验表明,该方法在提升基线性能的同时,还实现了超越准确率的优良特性,有望为时序分析中的可解释知识蒸馏建立新范式。

原文摘要 · Abstract (English)

Knowledge distillation has proven effective for model compression by transferring knowledge from a larger network called the teacher to a smaller network called the student. Current knowledge distillation in time series is predominantly based on logit and feature aligning techniques originally developed for computer vision tasks. These methods do not explicitly account for temporal data and fall short in two key aspects. First, the mechanisms by which the transferred knowledge helps the student model learning process remain unclear due to uninterpretability of logits and features. Second, these methods transfer only limited knowledge, primarily replicating the teacher predictive accuracy. As a result, student models often produce predictive distributions that differ significantly from those of their teachers, hindering their safe substitution for teacher models. In this work, we propose transferring interpretable knowledge by extending conventional logit transfer to convey not just the right prediction but also the right reasoning of the teacher. Specifically, we induce other useful knowledge from the teacher logits termed temporal saliency which captures the importance of each input timestep to the teacher prediction. By training the student with Temporal Saliency Distillation we encourage it to make predictions based on the same input features as the teacher. Temporal Saliency Distillation requires no additional parameters or architecture specific assumptions. We demonstrate that Temporal Saliency Distillation effectively improves the performance of baseline methods while also achieving desirable properties beyond predictive accuracy. We hope our work establishes a new paradigm for interpretable knowledge distillation in time series analysis.

知识蒸馏时序分析可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。