通过自适应奇异值与专家混合对齐，让LoRA逼近全量微调效果。
Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment
- 用SVD结构的专家混合动态整合先验知识
- 引入理论缩放因子使LoRA MoE性能接近全量微调
- 适用于需高效微调大模型的场景
低秩适配（LoRA）虽能实现大语言模型的参数高效微调，但性能常落后于全量微调（Full FT）。现有方法通过静态奇异值分解（SVD）子集初始化LoRA，难以充分挖掘预训练知识。另一路径是引入专家混合（MoE）架构，但权重错位与复杂梯度动态使在LoRA前应用SVD困难。为此，我们提出GOAT框架：(1) 采用SVD结构的MoE自适应整合相关先验；(2) 通过推导理论缩放因子，使优化过程与全量微调的MoE对齐。实验表明，仅通过合理缩放即可显著提升LoRA MoE的效率与性能。在25个数据集上验证，涵盖自然语言理解、常识推理、图像分类与自然语言生成，其表现达到当前最优，缩小了与全量微调的差距。
原文摘要 · Abstract (English)
While Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning for Large Language Models (LLMs), its performance often falls short of Full Fine-Tuning (Full FT). Current methods optimize LoRA by initializing with static singular value decomposition (SVD) subsets, leading to suboptimal leveraging of pre-trained knowledge. Another path for improving LoRA is incorporating a Mixture-of-Experts (MoE) architecture. However, weight misalignment and complex gradient dynamics make it challenging to adopt SVD prior to the LoRA MoE architecture. To mitigate these issues, we propose \underline{G}reat L\underline{o}R\underline{A} Mixture-of-Exper\underline{t} (GOAT), a framework that (1) adaptively integrates relevant priors using an SVD-structured MoE, and (2) aligns optimization with full fine-tuned MoE by deriving a theoretical scaling factor. We demonstrate that proper scaling, without modifying the architecture or training algorithms, boosts LoRA MoE's efficiency and performance. Experiments across 25 datasets, including natural language understanding, commonsense reasoning, image classification, and natural language generation, demonstrate GOAT's state-of-the-art performance, closing the gap with Full FT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。