用低秩适配提升大模型知识蒸馏效率，压缩更优。
LLM-NEO: Parameter Efficient Knowledge Distillation for Large Language Models
- 将低秩适配融入知识蒸馏，实现参数高效压缩。
- 在Llama 2和Llama 3.2上性能超越多个基线方法。
- 对多种LoRA变体鲁棒，适合资源受限场景部署。
知识蒸馏（KD）是压缩大语言模型（LLM）的主流方法。本文重新审视了KD与低秩适配（LoRA），发现二者遵循相同范式。受此启发，提出参数高效的蒸馏方法LLM-NEO，将LoRA引入KD以提升知识迁移效率。进一步总结了LLM-NEO的超参数调优指南。在压缩Llama 2和Llama 3.2的实验中，LLM-NEO显著优于多种基线方法。深入分析表明，该方法对不同LoRA变体具有强鲁棒性。代码与训练模型已开源于GitHub。
原文摘要 · Abstract (English)
Knowledge distillation (KD) has been a predominant method for compressing Large Language Models (LLMs). In this paper, we first revisit KD and Low-Rank Adaption (LoRA) and demonstrate that they follow the same paradigm. Inspired by this observation, we propose a parameter-efficient knowledge distillation method, LLM-NEO, which integrates LoRA into KD to improve the efficiency of knowledge transfer. After that, we summarize some valuable guidelines for the hyperparameters in LLM-NEO. Experimental results on compressing Llama 2 and Llama 3.2 show that LLM-NEO outperforms various baselines. Further analysis demonstrates the robustness of the proposed LLM-NEO on variants of LoRA. The code and trained models are available at [Github](https://github.com/yang3121099/LLM-Neo).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。