通过层间共享秩提升LoRA表达能力，代码与数学任务表现更优
RaSA: Rank-Sharing Low-Rank Adaptation
- 构建共享秩池，各层按需分配，不增加参数量
- 在CodeXGLUE和GSM8K上分别提升2.1%和3.7%准确率
- 适合需要高效微调大模型的开发者与研究者
低秩适应（LoRA）被广泛用于大语言模型的参数高效微调。然而，由于低秩约束带来的表达能力有限，其在代码生成和数学推理等严格任务中表现受限。为此，我们提出一种创新方法——秩共享低秩适应（RaSA），通过跨层部分秩共享，增强LoRA的表达能力。通过建立共享秩池并施加层特定权重，RaSA在不增加参数开销的前提下有效提升了秩的数量。该方法理论严谨且实验验证充分，不仅保持了LoRA的核心优势，还在代码与数学等挑战性任务中显著提升性能。代码、数据与脚本已开源：https://github.com/zwhe99/RaSA。
原文摘要 · Abstract (English)
Low-rank adaptation (LoRA) has been prominently employed for parameter-efficient fine-tuning of large language models (LLMs). However, the limited expressive capacity of LoRA, stemming from the low-rank constraint, has been recognized as a bottleneck, particularly in rigorous tasks like code generation and mathematical reasoning. To address this limitation, we introduce Rank-Sharing Low-Rank Adaptation (RaSA), an innovative extension that enhances the expressive capacity of LoRA by leveraging partial rank sharing across layers. By forming a shared rank pool and applying layer-specific weighting, RaSA effectively increases the number of ranks without augmenting parameter overhead. Our theoretically grounded and empirically validated approach demonstrates that RaSA not only maintains the core advantages of LoRA but also significantly boosts performance in challenging code and math tasks. Code, data and scripts are available at: https://github.com/zwhe99/RaSA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。