用密集低秩矩阵提升大模型微调效率，参数少80倍仍更准
DenseLoRA: Dense Low-Rank Adaptation of Large Language Models
- 用单个编码器-解码器压缩表示，替换传统两矩阵冗余结构
- 仅用0.01%参数达83.8%准确率，比LoRA省99%参数且更优
- 适合资源受限场景下的高效微调，尤其适合大模型轻量化部署
低秩适配（LoRA）通过微调两个低秩矩阵实现大语言模型的高效适配，显著减少可训练参数。然而，先前研究发现这些矩阵中存在大量冗余权重，导致参数利用效率低下。为此，我们提出密集低秩适配（DenseLoRA），一种在保持高效率的同时性能更优的新方法。DenseLoRA基于表征微调思想，引入单一编码器-解码器，在所有适配层前对隐藏表示进行精炼与压缩，再执行适配。相比LoRA依赖两个冗余低秩矩阵，DenseLoRA通过一个密集低秩矩阵实现模型适配，提升参数利用率与适配效率。我们在多个基准上评估该方法，结果显示，在LLaMA3-8B上，DenseLoRA仅使用0.01%可训练参数即达到83.8%准确率，而LoRA在使用0.70%参数时仅达80.8%。我们还进行了系统性实验，分析各组件对整体性能的影响。代码已开源。
原文摘要 · Abstract (English)
Low-rank adaptation (LoRA) has been developed as an efficient approach for adapting large language models (LLMs) by fine-tuning two low-rank matrices, thereby reducing the number of trainable parameters. However, prior research indicates that many of the weights in these matrices are redundant, leading to inefficiencies in parameter utilization. To address this limitation, we introduce Dense Low-Rank Adaptation (DenseLoRA), a novel approach that enhances parameter efficiency while achieving superior performance compared to LoRA. DenseLoRA builds upon the concept of representation fine-tuning, incorporating a single Encoder-Decoder to refine and compress hidden representations across all adaptation layers before applying adaptation. Instead of relying on two redundant low-rank matrices as in LoRA, DenseLoRA adapts LLMs through a dense low-rank matrix, improving parameter utilization and adaptation efficiency. We evaluate DenseLoRA on various benchmarks, showing that it achieves 83.8% accuracy with only 0.01% of trainable parameters, compared to LoRA's 80.8% accuracy with 0.70% of trainable parameters on LLaMA3-8B. Additionally, we conduct extensive experiments to systematically assess the impact of DenseLoRA's components on overall model performance. Code is available at https://github.com/mulin-ahu/DenseLoRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。