MALoRA通过异构低秩适配提升多任务学习效率与稳定性。
MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learning
- 采用不对称低秩适配设计，动态分配专家参数以优化训练
- 参数量减少30%~48%，训练速度提升1.2倍，媲美单任务效率
- 缓解高秩配置过拟合问题，适合资源受限的多任务场景
参数高效微调（PEFT）方法如LoRA显著提升了大模型在下游任务中的适应性。但在多任务场景中，训练不平衡和跷跷板效应常导致性能下降。混合专家型LoRA（MoLoRA）虽通过稀疏专家机制促进任务特异性学习，但存在训练速度慢、参数利用率低及整体性能不足的问题。本文提出混合异构低秩适配（MALoRA），通过在不同专家间实施非对称优化，将可训练参数减少30%至48%，训练速度提升1.2倍，计算效率接近单任务LoRA模型。同时，该方法有效缓解高秩配置下的过拟合问题，增强性能稳定性。在多种多任务学习场景下，包括跨领域与同领域任务，实验表明MALoRA始终优于所有基线方法。
原文摘要 · Abstract (English)
Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA have significantly improved the adaptation of LLMs to downstream tasks in a resource-efficient manner. However, in multi-task scenarios, challenges such as training imbalance and the seesaw effect frequently emerge. Mixture-of-LoRA (MoLoRA), which combines LoRA with sparse Mixture-of-Experts, mitigates some of these issues by promoting task-specific learning across experts. Despite this, MoLoRA remains inefficient in terms of training speed, parameter utilization, and overall multi-task performance. In this paper, we propose Mixture of Asymmetric Low-Rank Adaptaion (MALoRA), a flexible fine-tuning framework that leverages asymmetric optimization across LoRA experts. MALoRA reduces the number of trainable parameters by 30% to 48%, increases training speed by 1.2x, and matches the computational efficiency of single-task LoRA models. Additionally, MALoRA addresses overfitting issues commonly seen in high-rank configurations, enhancing performance stability. Extensive experiments across diverse multi-task learning scenarios demonstrate that MALoRA consistently outperforms all baseline methods in both inter-domain and intra-domain tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。