用张量分解让小模型变大，提升知识蒸馏效果且不增加推理开销
Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation
- 通过张量分解将小模型参数扩展为高维张量，实现过参数化
- 在视觉与自然语言任务中显著提升蒸馏性能，精度优于基线
- 适合需要高性能小模型的部署场景，如移动端与边缘计算
更大参数量的预训练模型在下游任务中表现优异，但其高昂的计算成本限制了实际应用。本文聚焦知识蒸馏（KD），旨在训练一个紧凑的学生模型以模仿大型教师模型。不同于以往工作，我们通过训练过程中的过参数化提升学生模型能力，而无需增加推理延迟。具体提出一种张量分解策略，将学生模型的参数矩阵高效、近乎无损地分解为高维张量，从而实现有效过参数化。为保证效率，进一步引入张量约束损失，使学生与教师模型间的高维张量对齐。大量实验验证了该方法在计算机视觉和自然语言处理任务中的显著性能提升。代码已开源：https://github.com/intell-sci-comput/OPDF。
原文摘要 · Abstract (English)
Increased training parameters have enabled large pre-trained models to excel in various downstream tasks. Nevertheless, the extensive computational requirements associated with these models hinder their widespread adoption within the community. We focus on Knowledge Distillation (KD), where a compact student model is trained to mimic a larger teacher model, facilitating the transfer of knowledge of large models. In contrast to much of the previous work, we scale up the parameters of the student model during training, to benefit from overparameterization without increasing the inference latency. In particular, we propose a tensor decomposition strategy that effectively over-parameterizes the relatively small student model through an efficient and nearly lossless decomposition of its parameter matrices into higher-dimensional tensors. To ensure efficiency, we further introduce a tensor constraint loss to align the high-dimensional tensors between the student and teacher models. Comprehensive experiments validate the significant performance enhancement by our approach in various KD tasks, covering computer vision and natural language processing areas. Our code is available at https://github.com/intell-sci-comput/OPDF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。