用多智能体教师蒸馏出更小更强的轻量大模型，适合资源受限场景。
DistilQwen2.5: Industrial Practices of Training Distilled Open Lightweight Language Models
- 用多个能力不同的私有大模型做教师，优化指令数据供学生学习
- 蒸馏后模型在指令遵循上显著优于原始版本
- 适合工业界部署，已开源全部模型供使用
在资源受限场景中，提升大语言模型的计算效率并降低部署成本已成为关键挑战。本文提出 DistilQwen2.5，是一系列从公开 Qwen2.5 模型蒸馏而来的轻量级 LLM 家族。通过融合多种蒸馏技术，这些模型从更大规模的 LLM 中吸收知识，在指令遵循能力上表现优于原始模型。在工业实践中，我们首先利用多个不同能力的私有大模型作为多智能体教师，对指令-响应对进行筛选、重写与精炼，使其更适于学生模型学习。经过标准微调后，进一步采用一种计算高效的模型融合方法，使学生模型逐步整合教师的细粒度隐藏知识。实验评估表明，蒸馏后的模型具备显著更强的能力。此外，我们展示了该框架在真实场景中的应用案例。为促进实际应用，所有 DistilQwen2.5 模型均已开源。
原文摘要 · Abstract (English)
Enhancing computational efficiency and reducing deployment costs for large language models (LLMs) have become critical challenges in various resource-constrained scenarios. In this work, we present DistilQwen2.5, a family of distilled, lightweight LLMs derived from the public Qwen2.5 models. These distilled models exhibit enhanced instruction-following capabilities compared to the original models based on a series of distillation techniques that incorporate knowledge from much larger LLMs. In our industrial practice, we first leverage powerful proprietary LLMs with varying capacities as multi-agent teachers to select, rewrite, and refine instruction-response pairs that are more suitable for student LLMs to learn. After standard fine-tuning, we further leverage a computationally efficient model fusion approach that enables student models to progressively integrate fine-grained hidden knowledge from their teachers. Experimental evaluations demonstrate that the distilled models possess significantly stronger capabilities than their original checkpoints. Additionally, we present use cases to illustrate the applications of our framework in real-world scenarios. To facilitate practical use, we have released all the DistilQwen2.5 models to the open-source community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。