对比不同学生模型下特征蒸馏的效果差异。
A Controlled Study of Feature-Based Knowledge Distillation Across Student Designs
- 在相同训练条件下，用多个学生模型测试特征蒸馏方法。
- 注意力迁移对不同模型影响不一，对MobileNetV2有益，对CustomResNet有害。
- 固定系数无法统一梯度尺度，说明需个性化调整蒸馏强度。
知识蒸馏通过让小型学生模型模仿大型教师模型的输出来提升性能。基于特征的方法还对中间表示进行对齐，但这种额外约束可能对不同学生模型产生不同影响。本研究在CIFAR-100上，使用ResNet-50作为教师模型，以宽度可控的CustomResNet家族和MobileNetV2作为跨设计对比的学生模型，评估了多种特征蒸馏方法。每个学生模型均在相同教师、优化器设置、训练策略和随机种子下，与对应的logit蒸馏（logit-KD）进行对比，并重复多组种子实验。结果显示，logit蒸馏在所有测试学生模型上均优于从头训练基线；注意力迁移在CustomResNet家族内部无明显大小相关性，但平均效果为负，而在MobileNetV2上为正；FitNets在全部15次配对实验中均低于logit蒸馏，且在宽度增加时差距扩大，但深度不同的宽48模型例外。此外，相同的辅助系数在不同学生间引发不同的梯度尺度，表明固定系数无法实现一致的训练条件。
原文摘要 · Abstract (English)
Knowledge distillation trains a smaller student to match the outputs of a larger teacher. Feature-based methods also align intermediate representations, but this extra constraint may affect students differently. We study this question on CIFAR-100 using a ResNet-50 teacher, a width-controlled CustomResNet family and MobileNetV2 as a cross-design comparison. For each student, we evaluate each feature method against a matched logit-KD run using the same teacher, optimizer settings, training schedule and seed. We repeat the main comparisons across multiple seeds. Logit KD improved every tested student over its scratch baseline. Attention Transfer showed no clear relationship with size inside the CustomResNet family, but its average effect was negative for that family and positive for MobileNetV2. FitNets was below logit KD in all 15 paired runs. Within the constant-depth width sweep, its gap increased for wider students, although the different-depth w=48 student did not follow this trend. Finally, the same auxiliary coefficient produced different gradient scales across students, showing that a fixed coefficient does not create a uniform training condition.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。