根据参数敏感性动态分配专家数量，提升小模型高效微调性能
A Sensitivity-Driven Expert Allocation Method in LoRA-MoE for Efficient Fine-Tuning
- 通过少量数据和梯度信息评估任务对参数的敏感性
- 在参数预算内自适应分配专家数，减少冗余参数30%以上
- 适合算力受限场景，计算开销极低，开源可复现
随着深度学习模型规模扩大，预训练-微调范式已成为处理下游任务的标准方法。然而，在涉及多任务的复杂数据集上，共享参数会导致性能下降。引入混合专家（MoE）虽缓解了该问题，却显著增加微调参数量和训练时间，带来更大参数冗余。为此，本文提出基于参数敏感性的专家数量分配方法——LoRA-SMoE。该方法通过采样少量数据并利用梯度信息快速评估不同任务对参数的敏感性，进而在给定预算下自适应分配专家数量。该过程保持与LoRA相当的内存消耗，同时实现高效、资源友好的微调。实验表明，相比现有最优微调方法，LoRA-SMoE在减少可训练参数的同时提升模型性能，显著改善资源受限环境下的表现。此外，其高效的参数敏感性评估机制使专家分配优化几乎无额外计算开销，特别适用于计算资源有限的场景。所有代码将在论文录用后公开，源码地址：https://github.com/EMLS-ICTCAS/LoRA-SMoE。
原文摘要 · Abstract (English)
As deep learning models expand, the pre-training-fine-tuning paradigm has become the standard approach for handling various downstream tasks. However, shared parameters can lead to diminished performance when dealing with complex datasets involving multiple tasks. While introducing Mixture-of-Experts (MoE) methods has alleviated this issue to some extent, it also significantly increases the number of parameters required for fine-tuning and training time, introducing greater parameter redundancy. To address these challenges, we propose a method for allocating expert numbers based on parameter sensitivity LoRA-SMoE (A Sensitivity-Driven Expert Allocation Method in LoRA-MoE for Efficient Fine-Tuning). This method rapidly assesses the sensitivity of different tasks to parameters by sampling a small amount of data and using gradient information. It then adaptively allocates expert numbers within a given budget. The process maintains comparable memory consumption to LoRA (Low-Rank Adaptation) while ensuring an efficient and resource-friendly fine-tuning procedure. Experimental results demonstrate that compared to SOTA fine-tuning methods, our LoRA-SMoE approach can enhance model performance while reducing the number of trainable parameters. This significantly improves model performance in resource-constrained environments. Additionally, due to its efficient parameter sensitivity evaluation mechanism, LoRA-SMoE requires minimal computational overhead to optimize expert allocation, making it particularly suitable for scenarios with limited computational resources. All the code in this study will be made publicly available following the acceptance of the paper for publication. Source code is at https://github.com/EMLS-ICTCAS/LoRA-SMoE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。