发现低层LoRA对推理更重要，可删掉高层以提速
Not All LoRA Parameters Are Essential: Insights on Inference Necessity
- 通过验证样本找关键层，只保留必要LoRA模块
- 在4个数据集上提升显著，最高加速30%
- 适合追求效率的LLM部署场景
当前关于LoRA的研究主要聚焦于减少微调参数数量或优化架构，但其所有微调层在推理阶段是否必要仍缺乏研究。本文探究每个LoRA层对模型预测真实输出的贡献，假设底层LoRA模块在模型推理与理解中起更关键作用。为此，提出一种简单有效的方法：通过分析少量验证样本，识别出区分关键LoRA层的“边界层”，推理时仅保留该层之前的所有模块。在三个强基线模型和四个广泛使用的文本生成数据集上评估，结果表明该方法带来持续且显著的性能提升，证明在推理阶段有选择性地保留核心LoRA层的有效性。
原文摘要 · Abstract (English)
Current research on LoRA primarily focuses on minimizing the number of fine-tuned parameters or optimizing its architecture. However, the necessity of all fine-tuned LoRA layers during inference remains underexplored. In this paper, we investigate the contribution of each LoRA layer to the model's ability to predict the ground truth and hypothesize that lower-layer LoRA modules play a more critical role in model reasoning and understanding. To address this, we propose a simple yet effective method to enhance the performance of large language models (LLMs) fine-tuned with LoRA. Specifically, we identify a ``boundary layer'' that distinguishes essential LoRA layers by analyzing a small set of validation samples. During inference, we drop all LoRA layers beyond this boundary. We evaluate our approach on three strong baselines across four widely-used text generation datasets. Our results demonstrate consistent and significant improvements, underscoring the effectiveness of selectively retaining critical LoRA layers during inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。