根据层训练质量动态分配LoRA专家，减少冗余提升效率
AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality
- 基于重尾自正则理论，按层训练质量分配专家数量
- 在10个任务上表现优于或媲美现有方法，无需额外训练
- 适合追求高效微调的LLM研究者与工程师
参数高效微调方法如低秩适配（LoRA）能显著提升大语言模型的训练效率。由于LoRA参数有限，近期研究尝试将其与专家混合（MoE）结合以提升性能。然而，受传统MoE结构冗余启发，已有研究发现LoRA专家在MoE中也存在类似冗余，表明应非均匀分配专家至不同层。本文基于重尾自正则化（HT-SR）理论，提出细粒度分配策略。分析显示，每层专家数量与该层训练质量相关，且层间差异显著。据此提出AlphaLoRA，一种理论驱动、无需训练的LoRA专家分配方法，可有效缓解冗余。在三个模型上对十个自然语言处理与推理基准的实验表明，AlphaLoRA性能优于或媲美所有基线。代码已公开于https://github.com/morelife2017/alphalora。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA), are known to enhance training efficiency in Large Language Models (LLMs). Due to the limited parameters of LoRA, recent studies seek to combine LoRA with Mixture-of-Experts (MoE) to boost performance across various tasks. However, inspired by the observed redundancy in traditional MoE structures, previous studies identify similar redundancy among LoRA experts within the MoE architecture, highlighting the necessity for non-uniform allocation of LoRA experts across different layers. In this paper, we leverage Heavy-Tailed Self-Regularization (HT-SR) Theory to design a fine-grained allocation strategy. Our analysis reveals that the number of experts per layer correlates with layer training quality, which exhibits significant variability across layers. Based on this, we introduce AlphaLoRA, a theoretically principled and training-free method for allocating LoRA experts to further mitigate redundancy. Experiments on three models across ten language processing and reasoning benchmarks demonstrate that AlphaLoRA achieves comparable or superior performance over all baselines. Our code is available at https://github.com/morelife2017/alphalora.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。