MSPLoRA通过分层低秩适配，让大模型微调更高效、参数更少。
MSPLoRA: A Multi-Scale Pyramid Low-Rank Adaptation for Efficient Model Fine-Tuning
- 分三层次设计低秩矩阵：全局共享、中层共享、逐层特异
- 在多个NLP任务上以更少参数实现更好性能，参数量减少显著
- 适合需要高效微调大模型的研究者和部署场景
参数高效微调（PEFT）已成为适应大规模预训练模型的重要方法，可降低计算成本。现有方法中，LoRA通过将权重更新分解为低秩矩阵来减少可训练参数。然而传统LoRA在各层采用固定秩，忽视了层次化信息的复杂性差异，导致适配效率低且冗余。为此，我们提出MSPLoRA（多尺度金字塔低秩适配），引入全局共享LoRA、中层共享LoRA和层特定LoRA，分别捕捉全局模式、中层特征与细粒度信息。该分层结构有效降低层间冗余，同时保持强适配能力。在多个自然语言处理任务上的实验表明，MSPLoRA实现了更高效的适配与更优性能，同时大幅减少可训练参数。基于奇异值分解的附加分析进一步验证其信息解耦能力，表明MSPLoRA是大语言模型参数高效微调中一种可扩展且有效的优化策略。代码已开源：https://github.com/Oblivioniss/MSPLoRA。
原文摘要 · Abstract (English)
Parameter-Efficient Fine-Tuning (PEFT) has become an essential approach for adapting large-scale pre-trained models while reducing computational costs. Among PEFT methods, LoRA significantly reduces trainable parameters by decomposing weight updates into low-rank matrices. However, traditional LoRA applies a fixed rank across all layers, failing to account for the varying complexity of hierarchical information, which leads to inefficient adaptation and redundancy. To address this, we propose MSPLoRA (Multi-Scale Pyramid LoRA), which introduces Global Shared LoRA, Mid-Level Shared LoRA, and Layer-Specific LoRA to capture global patterns, mid-level features, and fine-grained information, respectively. This hierarchical structure reduces inter-layer redundancy while maintaining strong adaptation capability. Experiments on various NLP tasks demonstrate that MSPLoRA achieves more efficient adaptation and better performance while significantly reducing the number of trainable parameters. Furthermore, additional analyses based on Singular Value Decomposition validate its information decoupling ability, highlighting MSPLoRA as a scalable and effective optimization strategy for parameter-efficient fine-tuning in large language models. Our code is available at https://github.com/Oblivioniss/MSPLoRA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。