arXiv:2510.12615cs.LGcs.AI2025-10被引 1

知识蒸馏实际是带负面偏移的数据依赖正则化,非可靠压缩机制。

A Functional Perspective on Knowledge Distillation in Neural Networks

  • 从功能角度拆解蒸馏过程,用控制实验分离知识转移机制
  • 22组实验表明知识转移有限且常伴随负面信息传递
  • 适合关注模型安全与蒸馏机理的从业者,不推荐盲目压缩

知识蒸馏在学生模型的准确率和损失上被视为压缩手段,但其功能影响仍不明确。本文从功能视角量化蒸馏的压缩能力与知识迁移效果,将压缩与结构简化分离,以深化对蒸馏的理解。采用受控实验设计与假设检验,结合随机蒸馏对照,分析不同数据模态下的知识转移机制。研究涵盖自蒸馏、标准蒸馏、特征图匹配变体、跨模型规模的蒸馏缩放规律及温度影响。在22个实验设置、9种架构和7个数据集上,发现部分模态与架构中存在统计显著的知识转移,但程度远低于预期,即使在最大化共享条件下亦然。值得注意的是,当功能迁移显著时,总存在一致且严重的负知识向学生模型的不对称传递,引发安全担忧。结果表明,知识蒸馏更像一种数据依赖的正则化,而非稳健的迁移压缩机制。

原文摘要 · Abstract (English)

Knowledge distillation is considered a compression mechanism when judged on the resulting student's accuracy and loss, yet its functional impact is poorly understood. We quantify the compression capacity of knowledge distillation and the resulting knowledge transfer from a functional perspective, decoupling compression from architectural reduction to provide an improved understanding of knowledge distillation. We employ a control-driven experimental protocol with hypothesis testing and random control distillation to isolate and understand knowledge transfer mechanisms across data modalities. To test the breadth and limits of our analyses, we study self-distillation, standard distillation, feature-map matching variants, distillation scaling laws across model sizes, and the impact of temperature on knowledge transfer. We find statistically supported knowledge transfer in some modalities and architectures; however, the extent of this transfer is less pronounced than anticipated, even under conditions that maximise knowledge sharing. Notably, in cases of significant functional transfer, we identify a consistent and severe asymmetric transfer of negative knowledge to the student, raising safety concerns for knowledge distillation. Across 22 experimental setups, 9 architectures, and 7 datasets, our results suggest that knowledge distillation functions less as a robust compression-by-transfer mechanism and more as a data-dependent regulariser whose transfer component is biased towards negative asymmetric transfer.

知识蒸馏模型压缩负知识功能分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。