解决多任务低秩适配扩展时性能暴跌问题，提升效率与准确性。
Scalable Multi-Task Low-Rank Model Adaptation
- 按频谱特性选择性正则化，保护共享知识成分。
- 块级参数适配降低梯度冲突，节省47%参数。
- 细粒度路由增强表达能力，适合大规模多任务场景。
将多任务低秩适配（LoRA）扩展到大量任务时，性能显著下降，例如在DOTA上任务数从5增至15，准确率由88.2%降至2.0%。根本原因在于参数与表征错位。现有方法如正则化和动态路由在规模扩大时失效，因存在根本权衡：强化正则化以减少任务间冲突，会抑制有效路由所需的关键特征区分能力。本文揭示两大根源：其一，均匀正则化破坏任务间知识共享——共享知识集中在高奇异值（SV）成分（在Flanv2→BBH上达89%对齐），均匀正则化强制高SV成分沿正交方向更新，直接干扰共享知识；其二，冲突放大：在组件级（如W_q、W_v）应用LoRA会放大梯度冲突，而块级适配可使冲突降低76%，仅需50%参数。基于此，提出mtLoRA，包含三项创新：1）谱感知正则化，选择性正交化低SV成分，保留高SV共享知识；2）块级适配，缓解冲突放大并大幅提高参数效率；3）基于维度的细粒度路由，提升表达能力。在四个大规模（15-25任务）基准（视觉：DOTA、iNat2018；NLP：Dolly-15k、BBH）上，mtLoRA在DOTA、iNat2018、Dolly-15k、BBH分别取得91.7%、81.5%、44.5%、38.5%准确率，优于当前最优方案平均2.3%，同时减少47%参数与24%训练时间。
原文摘要 · Abstract (English)
Scaling multi-task low-rank adaptation (LoRA) to a large number of tasks induces catastrophic performance degradation, such as an accuracy drop from 88.2% to 2.0% on DOTA when scaling from 5 to 15 tasks. This failure is due to parameter and representation misalignment. We find that existing solutions, like regularization and dynamic routing, fail at scale because they are constrained by a fundamental trade-off: strengthening regularization to reduce inter-task conflict inadvertently suppresses the essential feature discrimination required for effective routing. In this work, we identify two root causes for this trade-off. First, uniform regularization disrupts inter-task knowledge sharing: shared underlying knowledge concentrates in high-SV components (89% alignment on Flanv2->BBH). Uniform regularization forces high-SV components to update in orthogonal directions, directly disrupting the shared knowledge. Second, Conflict Amplification: Applying LoRA at the component-level (e.g., W_q, W_v) amplifies gradient conflicts; we show block-level adaptation reduces this conflict by 76% with only 50% parameters. Based on these insights, we propose mtLoRA, a scalable solution with three novel designs: 1) Spectral-Aware Regularization to selectively orthogonalize low-SV components while preserving high-SV shared knowledge, 2) Block-Level Adaptation to mitigate conflict amplification and largely improve parameter efficiency, and 3) Fine-Grained Routing using dimension-specific weights for superior expressive power. On four large-scale (15-25 tasks) vision (DOTA and iNat2018) and NLP (Dolly-15k and BBH) benchmarks, mtLoRA achieves 91.7%, 81.5%, 44.5% and 38.5% accuracy on DOTA, iNat2018, Dolly-15k and BBH respectively, outperforming the state-of-the-art by 2.3% on average while using 47% fewer parameters and 24% less training time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。