通过共享权重与丢弃策略，实现高效多任务微调。
MoSLD: An Extremely Parameter-Efficient Mixture-of-Shared LoRAs for Multi-Task Learning
- 共享上层投影矩阵，融合跨任务通用知识。
- 在多个数据集上表现优于传统LoRA，泛化能力强。
- 参数量极低，适合资源受限场景的多任务学习。
近期，LoRA已成为微调大型预训练模型的关键技术，但在多任务学习中表现常不理想。相比之下，MoE架构虽具天然优势，却面临跨领域数据干扰与任务知识遗忘问题，且显著增加参数量，带来计算负担。为此，本文提出MoSLD——一种基于丢弃策略的共享式LoRAs混合模型。该模型通过在不同专家间共享LoRA的上层投影矩阵，促进模型学习跨任务通用知识，同时保留下层投影矩阵对各任务特性的专注。引入丢弃机制缓解参数矩阵更新失衡，减轻LoRA中的参数过拟合。大量实验表明，该模型在单任务与多任务场景下均表现出色，具备强域外泛化能力。
原文摘要 · Abstract (English)
Recently, LoRA has emerged as a crucial technique for fine-tuning large pre-trained models, yet its performance in multi-task learning scenarios often falls short. In contrast, the MoE architecture presents a natural solution to this issue. However, it introduces challenges such as mutual interference of data across multiple domains and knowledge forgetting of various tasks. Additionally, MoE significantly increases the number of parameters, posing a computational cost challenge. Therefore, in this paper, we propose MoSLD, a mixture-of-shared-LoRAs model with a dropout strategy. MoSLD addresses these challenges by sharing the upper projection matrix in LoRA among different experts, encouraging the model to learn general knowledge across tasks, while still allowing the lower projection matrix to focus on the unique features of each task. The application of dropout alleviates the imbalanced update of parameter matrix and mitigates parameter overfitting in LoRA. Extensive experiments demonstrate that our model exhibits excellent performance in both single-task and multi-task scenarios, with robust out-of-domain generalization capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。