提出正交梯度投影方法，解决多任务LoRA中的冲突问题。
Disentangling Task Conflicts in Multi-Task LoRA via Orthogonal Gradient Projection
- 在LoRA子空间内动态投影冲突梯度，保持任务独立性。
- 在GLUE上恢复95%单任务性能差距，显著提升多任务效果。
- 无需额外计算开销，适合高效部署大模型多任务应用。
多任务学习结合低秩适配（LoRA）已成为高效部署大语言模型的有前景方向。通过共享单一适配器实现多任务，可大幅降低存储开销。然而，不同任务间的梯度冲突会导致性能下降，尤其在低秩约束下更严重。本文提出Ortho-LoRA，一种针对LoRA双分图结构设计的梯度投影方法。该方法在内在的LoRA子空间中，将相互冲突的任务梯度动态投影到彼此的正交补空间。在GLUE基准上的大量实验表明，Ortho-LoRA有效缓解了任务干扰，优于标准联合训练，在几乎无额外计算开销下，恢复了95%的多任务与单任务基线之间的性能差距。
原文摘要 · Abstract (English)
Multi-Task Learning (MTL) combined with Low-Rank Adaptation (LoRA) has emerged as a promising direction for parameter-efficient deployment of Large Language Models (LLMs). By sharing a single adapter across multiple tasks, one can significantly reduce storage overhead. However, this approach suffers from negative transfer, where conflicting gradient updates from distinct tasks degrade the performance of individual tasks compared to single-task fine-tuning. This problem is exacerbated in LoRA due to the low-rank constraint, which limits the optimization landscape's capacity to accommodate diverse task requirements. In this paper, we propose Ortho-LoRA, a gradient projection method specifically tailored for the bipartite structure of LoRA. Ortho-LoRA dynamically projects conflicting task gradients onto the orthogonal complement of each other within the intrinsic LoRA subspace. Extensive experiments on the GLUE benchmark demonstrate that Ortho-LoRA effectively mitigates task interference, outperforming standard joint training and recovering 95\% of the performance gap between multi-task and single-task baselines with negligible computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。