通过动态剪枝提升LoRA性能，零成本实现更优微调效果。
DropLoRA: Sparse Low-Rank Adaptation for Parameter-Efficient Fine-Tuning
- 在LoRA低秩矩阵间加入剪枝模块，实现动态子空间学习。
- 在多个LLaMA任务上超越传统LoRA，如数学推理和代码生成。
- 无需额外训练或推理开销，适合资源受限场景的高效微调。
基于LoRA的大模型参数高效微调方法利用低秩分解近似参数更新，但相比全参数微调，常导致下游任务性能下降。为此，本文提出DropLoRA，一种基于剪枝的新型方法，专注于剪枝秩维度。与传统方法试图克服低秩瓶颈不同，DropLoRA在LoRA两个低秩矩阵间创新性引入剪枝模块，模拟动态子空间学习。该动态低秩子空间学习使模型突破传统LoRA静态子空间的限制，通过持续自适应学习子空间，显著提升性能且不增加训练或推理开销。实验表明，DropLoRA在广泛的大语言模型生成任务中，包括常识推理、数学推理、代码生成和指令遵循，均一致优于标准LoRA。代码已开源：https://github.com/TayeeChang/DropLoRA。
原文摘要 · Abstract (English)
LoRA-based large model parameter-efficient fine-tuning (PEFT) methods use low-rank de- composition to approximate updates to model parameters. However, compared to full- parameter fine-tuning, low-rank updates often lead to a performance gap in downstream tasks. To address this, we introduce DropLoRA, a novel pruning-based approach that focuses on pruning the rank dimension. Unlike conven- tional methods that attempt to overcome the low-rank bottleneck, DropLoRA innovatively integrates a pruning module between the two low-rank matrices in LoRA to simulate dy- namic subspace learning. This dynamic low- rank subspace learning allows DropLoRA to overcome the limitations of traditional LoRA, which operates within a static subspace. By continuously adapting the learning subspace, DropLoRA significantly boosts performance without incurring additional training or infer- ence costs. Our experimental results demon- strate that DropLoRA consistently outperforms LoRA in fine-tuning the LLaMA series across a wide range of large language model gener- ation tasks, including commonsense reason- ing, mathematical reasoning, code generation, and instruction-following. Our code is avail- able at https://github.com/TayeeChang/DropLoRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。