将LoRA与知识蒸馏结合，实现更小模型、更低显存的高效微调。
KD-LoRA: A Hybrid Approach to Efficient Fine-Tuning with LoRA and Knowledge Distillation
- 用知识蒸馏压缩LoRA微调后的模型，保留核心能力。
- 在GLUE上性能达LoRA的98%，模型体积缩小40%。
- 适合资源受限场景，如移动端部署或大规模模型迭代。
大语言模型在下游任务中表现卓越，但其高计算与内存需求成为主要瓶颈。为解决此问题，参数高效微调（PEFT）方法如低秩适配（LoRA）被提出，以降低计算成本并保持性能。此外，知识蒸馏（KD）常用于从教师模型生成紧凑的学生模型。本文提出KD-LoRA，一种结合LoRA与知识蒸馏的新微调方法。实验表明，KD-LoRA在保持与全量微调（FFT）和LoRA相当性能的同时，显著降低资源消耗。具体而言，其在GLUE基准上保留了LoRA 98%的性能，模型体积减少40%；相比LoRA，GPU显存使用降低30%，推理时间也减少30%。我们在三个仅编码器模型（BERT、RoBERTa、DeBERTaV3）上进行了评估。代码已开源：https://github.com/rambodazimi/KD-LoRA。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated remarkable performance across various downstream tasks. However, the high computational and memory requirements of LLMs are a major bottleneck. To address this, parameter-efficient fine-tuning (PEFT) methods such as low-rank adaptation (LoRA) have been proposed to reduce computational costs while ensuring minimal loss in performance. Additionally, knowledge distillation (KD) has been a popular choice for obtaining compact student models from teacher models. In this work, we present KD-LoRA, a novel fine-tuning method that combines LoRA with KD. Our results demonstrate that KD-LoRA achieves performance comparable to full fine-tuning (FFT) and LoRA while significantly reducing resource requirements. Specifically, KD-LoRA retains 98% of LoRA's performance on the GLUE benchmark, while being 40% more compact. Additionally, KD-LoRA reduces GPU memory usage by 30% compared to LoRA, while decreasing inference time by 30% compared to both FFT and LoRA. We evaluate KD-LoRA across three encoder-only models: BERT, RoBERTa, and DeBERTaV3. Code is available at https://github.com/rambodazimi/KD-LoRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。