通过分步逼近提升知识蒸馏效果，显著缩小师生模型差距。
Expandable Residual Approximation for Knowledge Distillation
- 将知识逼近分解为多步，降低学生模型模仿难度。
- 在ImageNet上提升1.41%准确率，在COCO上提升1.40%平均精度。
- 适合追求高精度轻量模型的视觉任务研究者使用。
知识蒸馏（KD）旨在将大型教师模型的知识迁移到轻量化学生模型，大幅降低计算与存储开销。然而，教师与学生模型间固有的学习能力差距常阻碍知识充分传递，促使众多研究应对此挑战。受Stone-Weierstrass定理中渐进逼近原理启发，我们提出可扩展残差逼近（ERA），一种新型知识蒸馏方法，将残差知识的逼近分解为多个步骤，通过分而治之策略降低学生模型模仿教师表征的难度。具体地，ERA采用多分支残差网络（MBRNet）实现残差知识分解，并引入教师权重融合（TWI）策略，通过复用教师模型头部权重缓解容量差异。大量实验表明，ERA在ImageNet图像分类基准上提升Top-1准确率1.41%，在MS COCO目标检测基准上提升平均精度（AP）1.40%，并在多项计算机视觉任务中达到领先性能。代码与模型已公开于https://github.com/Zhaoyi-Yan/ERA。
原文摘要 · Abstract (English)
Knowledge distillation (KD) aims to transfer knowledge from a large-scale teacher model to a lightweight one, significantly reducing computational and storage requirements. However, the inherent learning capacity gap between the teacher and student often hinders the sufficient transfer of knowledge, motivating numerous studies to address this challenge. Inspired by the progressive approximation principle in the Stone-Weierstrass theorem, we propose Expandable Residual Approximation (ERA), a novel KD method that decomposes the approximation of residual knowledge into multiple steps, reducing the difficulty of mimicking the teacher's representation through a divide-and-conquer approach. Specifically, ERA employs a Multi-Branched Residual Network (MBRNet) to implement this residual knowledge decomposition. Additionally, a Teacher Weight Integration (TWI) strategy is introduced to mitigate the capacity disparity by reusing the teacher's head weights. Extensive experiments show that ERA improves the Top-1 accuracy on the ImageNet classification benchmark by 1.41% and the AP on the MS COCO object detection benchmark by 1.40, as well as achieving leading performance across computer vision tasks. Codes and models are available at https://github.com/Zhaoyi-Yan/ERA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。