只蒸馏模型中与任务相关的部分，让小模型更高效准确
Distilling Lightweight Domain Experts from Large ML Models by Identifying Relevant Subspaces
- 通过识别关键子空间，仅蒸馏教师模型的相关部分
- 在CIFAR-100和ImageNet上优于现有层间蒸馏方法
- 适合资源受限场景下精准迁移特定类别知识
知识蒸馏将大型高性能AI模型（教师）的预测能力迁移到计算资源有限环境中的小型模型（学生）。本文针对仅少数类别及其相关中间概念重要的实际场景，提出新算法SubDistill。该方法具有更好的数值性质，仅在每一层蒸馏教师模型的相关组件。在使用卷积和Transformer模型的CIFAR-100与ImageNet数据集上的实验表明，SubDistill在代表性子任务中表现优于现有层间蒸馏技术。基准评估结合可解释AI分析显示，蒸馏后的学生模型更贴近原始教师模型的决策结构。
原文摘要 · Abstract (English)
Knowledge distillation involves transferring the predictive capabilities of large, high-performing AI models (teachers) to smaller models (students) that can operate in environments with limited computing power. In this paper, we address the scenario in which only a few classes and their associated intermediate concepts are relevant to distill. This scenario is common in practice, yet few existing distillation methods explicitly focus on the relevant subtask. To address this gap, we introduce 'SubDistill', a new distillation algorithm with improved numerical properties that only distills the relevant components of the teacher model at each layer. Experiments on CIFAR-100 and ImageNet with Convolutional and Transformer models demonstrate that SubDistill outperforms existing layer-wise distillation techniques on a representative set of subtasks. Our benchmark evaluations are complemented by Explainable AI analyses showing that our distilled student models more closely match the decision structure of the original teacher model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。