用定制坐标系让小模型高效学大模型的隐含知识
All You Need in Knowledge Distillation Is a Tailored Coordinate System
- 用预训练模型特征空间的坐标系做知识蒸馏，无需专门训练教师模型
- 仅需一次教师前向计算，训练时间与显存消耗减半
- 适合跨架构小样本学习，对不同模型结构兼容性好
知识蒸馏（KD）是将大模型中的暗知识迁移到小模型的有效方法，使小模型在保持高精度的同时显著提升效率。然而现有方法依赖为特定任务专门训练的大教师模型，灵活性和效率均受限。本文提出，自监督预训练模型可作为有效教师，其暗知识可通过特征所在坐标系或线性子空间捕捉。只需一次教师前向传播，即可为学生网络定制坐标系统（TCS）。该方法无需教师模型，适用于多种网络架构，在知识蒸馏和实际少样本学习中表现优异，并支持大容量差距的跨架构蒸馏。实验表明，TCS在准确率上显著优于当前最优方法，训练时间与GPU显存消耗仅为其一半。
原文摘要 · Abstract (English)
Knowledge Distillation (KD) is essential in transferring dark knowledge from a large teacher to a small student network, such that the student can be much more efficient than the teacher but with comparable accuracy. Existing KD methods, however, rely on a large teacher trained specifically for the target task, which is both very inflexible and inefficient. In this paper, we argue that a SSL-pretrained model can effectively act as the teacher and its dark knowledge can be captured by the coordinate system or linear subspace where the features lie in. We then need only one forward pass of the teacher, and then tailor the coordinate system (TCS) for the student network. Our TCS method is teacher-free and applies to diverse architectures, works well for KD and practical few-shot learning, and allows cross-architecture distillation with large capacity gap. Experiments show that TCS achieves significantly higher accuracy than state-of-the-art KD methods, while only requiring roughly half of their training time and GPU memory costs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。