通过专家混合视角重参数化LoRA，显著提升微调效率
RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Experts
- 将LoRA重构为专家混合框架，用轻量MLP重新参数化矩阵
- 理论证明数据需求从指数级降至多项式级，误差控制更优
- 在少样本下性能领先40%，仅需30%数据即达原LoRA效果
低秩适配（LoRA）已成为大规模基础模型微调的有效方法。尽管广泛应用，其理论理解仍有限。本文通过专家混合模型视角分析LoRA，证明对LoRA矩阵进行简单重参数化可显著加速低秩矩阵估计过程。理论上，重参数化将达到指定估计误差所需的数据量从指数级降至多项式级。基于此，我们提出重参数化低秩适配（RepLoRA），引入轻量MLP对LoRA矩阵进行重参数化。跨多个领域的大量实验表明，RepLoRA始终优于原始LoRA。尤其在数据有限时，其性能最高超出LoRA 40.0%，且仅需30.0%的训练数据即可达到LoRA的性能水平，充分展现了该参数高效微调方法的理论与实证鲁棒性。
原文摘要 · Abstract (English)
Low-rank Adaptation (LoRA) has emerged as a powerful method for fine-tuning large-scale foundation models. Despite its popularity, the theoretical understanding of LoRA has remained limited. This paper presents a theoretical analysis of LoRA by examining its connection to the Mixture of Experts models. Under this framework, we show that simple reparameterizations of the LoRA matrices can notably accelerate the low-rank matrix estimation process. In particular, we prove that reparameterization can reduce the data needed to achieve a desired estimation error from an exponential to a polynomial scale. Motivated by this insight, we propose Reparameterized Low-Rank Adaptation (RepLoRA), which incorporates lightweight MLPs to reparameterize the LoRA matrices. Extensive experiments across multiple domains demonstrate that RepLoRA consistently outperforms vanilla LoRA. Notably, with limited data, RepLoRA surpasses LoRA by a margin of up to 40.0% and achieves LoRA's performance with only 30.0% of the training data, highlighting both the theoretical and empirical robustness of our PEFT method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。