通过逐层评估重要性，高效压缩大模型并保持性能。
Iterative Layer-wise Distillation for Efficient Compression of Large Language Models
- 逐层测试删减影响，动态识别可移除的低效层。
- 36层模型压缩至24层，仅损失18%性能，参数降至24.7亿。
- 适合边缘设备部署，尤其关注模型轻量化的人群。
本文研究大语言模型（LLM）的蒸馏压缩方法，目标是构建高性能的紧凑模型。通过分析现有方法的优劣，提出基于ShortGPT改进的迭代式分层蒸馏策略，结合逐层重要性评估：在每一步中，通过代表性数据集测量删除单个层导致的性能下降，以此判断其重要性。随后采用基于KL散度和均方误差的联合损失函数进行微调。在Qwen2.5-3B模型上的实验表明，层数可从36层缩减至28层（参数降至24.7亿），性能损失仅9.7%；进一步压缩至24层时，性能损失为18%。结果表明中间的Transformer层对推理贡献较小，验证了该方法在构建高效模型方面的潜力。实验充分证明迭代蒸馏与微调的有效性，适用于资源受限场景下的模型部署。
原文摘要 · Abstract (English)
This work investigates distillation methods for large language models (LLMs) with the goal of developing compact models that preserve high performance. Several existing approaches are reviewed, with a discussion of their respective strengths and limitations. An improved method based on the ShortGPT approach has been developed, building upon the idea of incorporating iterative evaluation of layer importance. At each step, importance is assessed by measuring performance degradation when individual layers are removed, using a set of representative datasets. This process is combined with further training using a joint loss function based on KL divergence and mean squared error. Experiments on the Qwen2.5-3B model show that the number of layers can be reduced from 36 to 28 (resulting in a 2.47 billion parameter model) with only a 9.7% quality loss, and to 24 layers with an 18% loss. The findings suggest that the middle transformer layers contribute less to inference, underscoring the potential of the proposed method for creating efficient models. The results demonstrate the effectiveness of iterative distillation and fine-tuning, making the approach suitable for deployment in resource-limited settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。