改进LoRA的特征瓶颈，用多个专家共享提升模型适应能力
MASA: Rethinking the Representational Bottleneck in LoRA with Multi-A Shared Adaptation
- 设计多下投影矩阵结构，让不同特征由多个专家分别提取
- 在MMLU上达59.62%准确率，比标准LoRA高1.08点
- 适合需要高效微调复杂任务的科研与工业应用
低秩适配（LoRA）是大语言模型参数高效微调的主流方法，通过一个下投影矩阵 $A$ 和一个上投影矩阵 $B$ 增强Transformer层。然而，其依赖单一 $A$ 矩阵导致表征瓶颈，难以捕捉复杂任务所需多样信号。为此，我们提出MASA（多$A$共享适配），采用多$A$、单$B$结构，让异构$A$专家跨层非对称共享以保持参数效率。各专家提取多样化特征，再由分层专属的$B$矩阵融合。实验覆盖多领域泛化、单领域专精及多任务推理，结果表明：在MMLU基准上，MASA达到平均59.62%准确率，较标准LoRA提升1.08个百分点（相对提升1.84%），且可学习参数量仅0.52%。
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) has emerged as a dominant method in Parameter-Efficient Fine-Tuning (PEFT) for large language models, which augments the transformer layer with one down-projection $A$ and one up-projection $B$. However, LoRA's reliance on a single down-projection matrix ($A$) creates a representational bottleneck, as this solitary feature extractor is inherently insufficient for capturing the diverse signals required by complex tasks. This motivates our architectural shift to focus on enriching the feature adaptation to improve the downstream task adaptation ability. We propose MASA (Multi-$A$ Shared Adaptation), an architecture that implements a multi-$A$, single-$B$ structure where the multi-$A$ expert ensemble is asymmetrically shared across layers to ensure parameter efficiency. In MASA, these specialized experts capture diverse features, which are then integrated by a single, layer-specific $B$-matrix. The effectiveness and versatility of our method are validated through a comprehensive suite of experiments spanning multi-domain generalization, single-domain specialization, and multi-task reasoning. For example, on the MMLU benchmark, MASA achieves an average accuracy of 59.62%, outperforming the standard LoRA by 1.08 points (a relative improvement of 1.84%) with comparable learnable parameters of 0.52%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。