用统一分布约束实现多阶段知识迁移，提升模型压缩效果
Harmonizing knowledge Transfer in Neural Network with Unified Distillation
- 将中间层特征融合为综合表征，捕捉多尺度语义信息
- 通过分布参数预测实现跨阶段知识传递，提升迁移完整性
- 适用于模型轻量化场景，尤其适合对精度要求高的部署
知识蒸馏（KD）因其能在不改变架构的前提下，将复杂网络（教师）的知识迁移到轻量网络（学生）中而受到广泛关注。现有方法主要分为基于特征的（关注中间层输出）和基于软标签的（关注最后一层输出）。本文提出一种新视角，在统一蒸馏框架下整合多种知识源：将中间层特征聚合为综合表征，有效获取不同阶段与尺度的语义信息；随后从该表征中预测分布参数，将中间层知识转化为对应的分布形式，从而在不同网络阶段施加统一的分布约束，确保知识迁移的全面性与一致性。大量实验验证了所提方法的有效性。
原文摘要 · Abstract (English)
Knowledge distillation (KD), known for its ability to transfer knowledge from a cumbersome network (teacher) to a lightweight one (student) without altering the architecture, has been garnering increasing attention. Two primary categories emerge within KD methods: feature-based, focusing on intermediate layers' features, and logits-based, targeting the final layer's logits. This paper introduces a novel perspective by leveraging diverse knowledge sources within a unified KD framework. Specifically, we aggregate features from intermediate layers into a comprehensive representation, effectively gathering semantic information from different stages and scales. Subsequently, we predict the distribution parameters from this representation. These steps transform knowledge from the intermediate layers into corresponding distributive forms, thereby allowing for knowledge distillation through a unified distribution constraint at different stages of the network, ensuring the comprehensiveness and coherence of knowledge transfer. Numerous experiments were conducted to validate the effectiveness of the proposed method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。