用流形映射蒸馏，让小模型逼近大模型性能
Logit Distillation on Manifolds: Mapping by Learning
- 在高维空间对齐师生模型表示，实现精准知识迁移
- 参数量降至教师模型1%以下，WER显著优于传统方法
- 训练快且可并行，适合大规模部署的轻量化模型
提升机器学习模型性能的常见方法是训练多个算法各异的模型，它们对相同数据产生略有不同的预测与误差，从而提高平均预测精度和鲁棒性。然而，使用整个模型集成进行预测计算开销大，难以大规模部署，尤其当模型为大型神经网络时。为此，我们提出一种逐层逐点的投影映射方法，在训练过程中将学生与教师模型的表示映射到对齐的高维嵌入空间。该方法结合LoRA注入，使学生模型可训练参数少于教师模型的1%，同时在消融实验中显著改善词错误率（WER）。与专家混合模型不同，本方法可快速且并行训练。
原文摘要 · Abstract (English)
A simple way to improve the performance of almost any machine learning model is not to train a single but several models with diverse algorithms which will make slightly distinct kinds of predictions and errors on the same data, and thus improve the average predictions and robustness. However, making predictions using a whole ensemble of models is cumbersome and computationally too expensive to allow deployment to a large number of users, especially if the models are large neural nets. In response to this, we introduce a layer and point wise projection mapping, which maps student and teacher representations into an aligned high-dimensional embedding space during training process. The proposed approach combined with LoRA injection reduces the student model trainable parameters to less than 1% of the teacher model, while significantly improving word error rate (WER) compared to other distillation methods, as demonstrated in ablation studies. Unlike a mixture of experts, our method can be trained rapidly and in parallel.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。