arXiv:2503.13008cs.LGcs.CV2025-03被引 4

用梯度热力图引导学生模型学习,让小模型更准更快。

Knowledge Distillation: Enhancing Neural Network Compression with Integrated Gradients

  • 用教师模型的梯度图叠加图像,指导学生模型聚焦关键特征。
  • 压缩4.1倍后准确率达92.5%,推理速度提升十倍至13毫秒。
  • 兼具高效与可解释性,适合对透明度要求高的边缘设备。

在资源受限设备上高效部署深度神经网络需先进压缩技术以保持精度和兼容性。本文提出一种机器学习框架,将知识蒸馏(KD)与积分梯度(IG)结合,优化卷积神经网络的压缩。我们引入一种新数据增强策略:预先计算教师模型的积分梯度图,并叠加到训练图像上,引导紧凑的学生模型关注关键特征表示。该方法利用教师的决策洞察,提升学生模型以较少参数复现复杂模式的能力。在CIFAR-10上的实验表明,学生模型压缩4.1倍后,分类准确率达92.5%,优于基线学生模型的91.4%及传统KD方法,推理延迟从140毫秒降至13毫秒(十倍加速)。通过超参数优化实现高效学习。全面的消融研究揭示了KD与IG的协同效应,同时提升性能与模型可解释性。本方法强调基于特征层面的IG引导,区别于传统KD,为神经架构中可迁移知识的挖掘提供数据驱动方案。该工作贡献了一种可扩展、可解释的压缩技术,适用于对效率与透明度要求极高的边缘计算场景。

原文摘要 · Abstract (English)

Efficient deployment of deep neural networks on resource-constrained devices demands advanced compression techniques that preserve accuracy and interoperability. This paper proposes a machine learning framework that augments Knowledge Distillation (KD) with Integrated Gradients (IG), an attribution method, to optimise the compression of convolutional neural networks. We introduce a novel data augmentation strategy where IG maps, precomputed from a teacher model, are overlaid onto training images to guide a compact student model toward critical feature representations. This approach leverages the teacher's decision-making insights, enhancing the student's ability to replicate complex patterns with reduced parameters. Experiments on CIFAR-10 demonstrate the efficacy of our method: a student model, compressed 4.1-fold from the MobileNet-V2 teacher, achieves 92.5% classification accuracy, surpassing the baseline student's 91.4% and traditional KD approaches, while reducing inference latency from 140 ms to 13 ms--a tenfold speedup. We perform hyperparameter optimisation for efficient learning. Comprehensive ablation studies dissect the contributions of KD and IG, revealing synergistic effects that boost both performance and model explainability. Our method's emphasis on feature-level guidance via IG distinguishes it from conventional KD, offering a data-driven solution for mining transferable knowledge in neural architectures. This work contributes to machine learning by providing a scalable, interpretable compression technique, ideal for edge computing applications where efficiency and transparency are paramount.

知识蒸馏模型压缩可解释性边缘计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。