通过跨层共享矩阵提升参数效率，实现更少参数更强性能。
ASLoRA: Adaptive Sharing Low-Rank Adaptation Across Layers
- 共享所有层的A矩阵，动态调整B矩阵以适应任务。
- 仅用不到25%参数，性能超越LoRA。
- 适合资源受限场景下的大模型微调。
随着大语言模型规模持续增长，传统全参数微调因计算与存储成本过高而变得不切实际。尽管像LoRA这样的参数高效微调方法已大幅减少可训练参数量，但仍存在优化空间。本文提出ASLoRA，一种结合全局共享与部分自适应共享的跨层参数共享策略：在所有层间共享低秩矩阵A，同时在训练过程中自适应融合矩阵B。该机制有效缓解过拟合，捕捉层间依赖关系，显著增强模型表征能力。在多个NLP任务上的实验表明，ASLoRA在使用少于25%参数的情况下优于LoRA，展现出卓越的参数效率。深入分析进一步验证了自适应共享策略在提升模型灵活性与任务适应性方面的显著优势。
原文摘要 · Abstract (English)
As large language models (LLMs) grow in size, traditional full fine-tuning becomes increasingly impractical due to its high computational and storage costs. Although popular parameter-efficient fine-tuning methods, such as LoRA, have significantly reduced the number of tunable parameters, there is still room for further optimization. In this work, we propose ASLoRA, a cross-layer parameter-sharing strategy combining global sharing with partial adaptive sharing. Specifically, we share the low-rank matrix A across all layers and adaptively merge matrix B during training. This sharing mechanism not only mitigates overfitting effectively but also captures inter-layer dependencies, significantly enhancing the model's representational capability. We conduct extensive experiments on various NLP tasks, showing that ASLoRA outperforms LoRA while using less than 25% of the parameters, highlighting its flexibility and superior parameter efficiency. Furthermore, in-depth analyses of the adaptive sharing strategy confirm its significant advantages in enhancing both model flexibility and task adaptability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。