动态调整知识蒸馏权重，提升小模型准确率与训练稳定性
Context-Aware Knowledge Distillation with Adaptive Weighting for Image Classification
- 让蒸馏权重α可学习，根据学生-教师差异自适应调整
- 在CIFAR-10上比固定权重基线提升准确率，收敛更稳定
- 引入上下文感知模块，对不同类别输出进行加权优化
知识蒸馏（KD）是将大模型知识迁移到小模型的常用技术。传统方法使用固定超参数α平衡硬标签交叉熵损失与软标签蒸馏损失，但静态α难以适应训练过程中的最优权衡。本文提出自适应知识蒸馏（AKD）框架：首先将α设为可学习参数，实现训练中自动优化；其次设计公式，基于学生-教师差异动态计算α；并引入上下文感知模块（CAM），通过MLP+注意力机制自适应重加权类别级教师输出。在以ResNet-50为教师、ResNet-18为学生模型的CIFAR-10实验中，所提方法优于固定权重基线，准确率更高且收敛更稳定。
原文摘要 · Abstract (English)
Knowledge distillation (KD) is a widely used technique to transfer knowledge from a large teacher network to a smaller student model. Traditional KD uses a fixed balancing factor alpha as a hyperparameter to combine the hard-label cross-entropy loss with the soft-label distillation loss. However, a static alpha is suboptimal because the optimal trade-off between hard and soft supervision can vary during training. In this work, we propose an Adaptive Knowledge Distillation (AKD) framework. First we try to make alpha as learnable parameter that can be automatically learned and optimized during training. Then we introduce a formula to reflect the gap between the student and the teacher to compute alpha dynamically, guided by student-teacher discrepancies, and further introduce a Context-Aware Module (CAM) using MLP + Attention to adaptively reweight class-wise teacher outputs. Experiments on CIFAR-10 with ResNet-50 as teacher and ResNet-18 as student demonstrate that our approach achieves superior accuracy compared to fixed-weight KD baselines, and yields more stable convergence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。