一套方法训练多个可部署的学生模型，兼顾精度与成本。
Matryoshka Model Learning for Improved Elastic Student Models
- 用更大容量的TA模型辅助学生模型学习，提升泛化能力。
- 单次训练可生成多个精度各异的学生模型，部署灵活。
- 实测关键指标提升20%，适合工业级模型快速迭代。
工业级机器学习模型需应对不断变化的服务约束，但开发成本高昂。本文提出MatTA框架，采用教师-塔-学生的新范式：塔（TA）模型是容量更大的学生模型变体，能引入更多领域知识并帮助学生模型更精准地逼近教师模型。通过该方法，仅一次训练即可提取多个高精度学生模型，实现精度与服务成本之间的灵活权衡。我们在自有数据集和模型上验证了该方法，生产系统中的实时A/B测试显示关键指标提升20%。此外，在GPT-2 Medium上应用，SAT Math相对提升超24%，LAMBADA基准提升超10%。
原文摘要 · Abstract (English)
Industry-grade ML models are carefully designed to meet rapidly evolving serving constraints, which requires significant resources for model development. In this paper, we propose MatTA, a framework for training multiple accurate Student models using a novel Teacher-TA-Student recipe. TA models are larger versions of the Student models with higher capacity, and thus allow Student models to better relate to the Teacher model and also bring in more domain-specific expertise. Furthermore, multiple accurate Student models can be extracted from the TA model. Therefore, despite only one training run, our methodology provides multiple servable options to trade off accuracy for lower serving cost. We demonstrate the proposed method, MatTA, on proprietary datasets and models. Its practical efficacy is underscored by live A/B tests within a production ML system, demonstrating 20% improvement on a key metric. We also demonstrate our method on GPT-2 Medium, a public model, and achieve relative improvements of over 24% on SAT Math and over 10% on the LAMBADA benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。