将多个模型的集体智慧压缩成一个更易部署的小模型，提升性能并降低计算开销。
Distilling the Knowledge in a Neural Network
- 用知识蒸馏技术将集成模型的知识压缩到单个模型中。
- 在MNIST上取得意外好结果，显著优化了商用语音系统性能。
- 提出新集成结构：主模型加多个专注细分类别的专家模型，训练快且可并行。
几乎任何机器学习算法的性能都可通过在相同数据上训练多个模型并平均其预测来提升。然而，使用整个模型集成进行预测既繁琐又可能因计算成本过高而难以大规模部署，尤其当个体模型为大型神经网络时。Caruana及其合作者已证明，可将集成模型中的知识压缩为单一模型，从而更易于部署，本文进一步采用另一种压缩方法加以改进。我们在MNIST数据集上取得了令人意外的良好结果，并证明通过将集成模型的知识蒸馏至单个模型，能显著提升一个广泛使用的商业语音系统的声学模型性能。此外，我们引入一种新型集成结构：由一个或多个全模型与多个专注于区分细粒度类别的专家模型组成。这些专家模型不同于混合专家架构，可快速且并行训练。
原文摘要 · Abstract (English)
A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。