蒸馏模型在资源受限场景下效率远超原始模型，80亿参数模型比原生模型节省2000倍算力。
Benchmarking Distilled Language Models: Performance and Efficiency in Resource-Constrained Settings
- 用知识蒸馏训练小模型，显著降低计算成本
- 80亿参数蒸馏模型算力效率超原生模型2000倍
- 性能媲美甚至超过十倍大小的标准模型，适合部署
知识蒸馏为在资源受限环境中构建强大且高效的轻量级语言模型（SLMs）提供了变革性路径。本文对蒸馏模型与原始及商业模型的性能和计算成本进行了基准测试，定量分析其效率。结果表明，蒸馏能实现更优的性能-计算权衡。我们发现，训练一个80亿参数的蒸馏模型，其计算效率比训练对应的原始模型高出2000多倍，同时在推理能力上达到或超越规模为其十倍的标准模型。这些发现验证了蒸馏不仅是压缩技术，更是构建前沿、可访问人工智能的核心策略。
原文摘要 · Abstract (English)
Knowledge distillation offers a transformative pathway to developing powerful, yet efficient, small language models (SLMs) suitable for resource-constrained environments. In this paper, we benchmark the performance and computational cost of distilled models against their vanilla and proprietary counterparts, providing a quantitative analysis of their efficiency. Our results demonstrate that distillation creates a superior performance-tocompute curve. We find that creating a distilled 8B model is over 2,000 times more compute-efficient than training its vanilla counterpart, while achieving reasoning capabilities on par with, or even exceeding, standard models ten times its size. These findings validate distillation not just as a compression technique, but as a primary strategy for building state-of-the-art, accessible AI
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。