通过互信息优化微调大模型，提升小模型知识蒸馏效果
ReMem: Mutual Information-Aware Fine-tuning of Pretrained Vision Transformers for Effective Knowledge Distillation
- 用互信息感知优化方法微调预训练ViT，增强知识迁移能力
- 在小数据或不平衡数据上，重加权MLP层使蒸馏效果提升显著
- 适合需要高效压缩大模型的工业级视觉任务应用
从预训练视觉表征模型中进行知识蒸馏是提升小型专用模型性能的有效方法。然而，当从大规模预训练的强模型中蒸馏时，知识传递效果会显著下降。本文针对预训练的视觉变换器（ViTs），提出通过互信息感知优化来微调模型以提升知识转移效率。针对小规模或高度不平衡的下游数据集，我们引入一种简单但有效的启发式方法：重加权MLP模块。该方法基于观察——顶层MLP模块主要导致互信息损失。本方法使小型学生模型能够有效利用最强的预训练模型。
原文摘要 · Abstract (English)
Knowledge distillation from pretrained visual representation models offers an effective approach to improve small, task-specific production models. However, the effectiveness of such knowledge transfer drops significantly when distilling from strong models that are pretrained in a large scale. In this paper, we address this challenge for pretrained Vision Transformers (ViTs) by exploring methods to fine-tune them for more effective knowledge transfer. Motivated by the connection between mutual information and distillation effectiveness, we propose to employ mutual information-aware optimization during finetuning. For small or highly-imbalanced downstream datasets where such optimization becomes less effective, we introduce a simple yet effective heuristic of reweighting MLP blocks. This approach is inspired by our observation that top MLP blocks are primarily responsible for mutual information loss. Our method enables small student models to benefit from those pretrained models among the strongest.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。