LoRI通过稀疏化与正交性设计，大幅减少多任务微调中的参数干扰。
LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation
- 冻结矩阵A为随机投影，用任务掩码稀疏化矩阵B
- 训练参数比LoRA少95%，多任务表现更优
- 适合需要持续学习和高效适配的场景
低秩适应（LoRA）是大型语言模型参数高效微调的热门方法，但在多任务场景中仍存在显著开销和参数干扰问题。本文提出降低干扰的低秩适应（LoRI），通过将投影矩阵 $A$ 冻结为随机投影，并使用任务特定掩码稀疏化矩阵 $B$，大幅减少可训练参数数量，同时保持强任务性能。此外，LoRI利用适配器子空间间的正交性，最小化适配器合并时的跨任务干扰，并通过稀疏性支持持续学习，缓解灾难性遗忘。在自然语言理解、数学推理、代码生成和安全对齐等任务上的广泛实验表明，LoRI优于全量微调和现有参数高效微调方法，且相比LoRA可减少高达95%的可训练参数。多任务实验显示，LoRI能有效实现适配器合并与持续学习，显著降低跨任务干扰。代码已开源：https://github.com/juzhengz/LoRI
原文摘要 · Abstract (English)
Low-Rank Adaptation (LoRA) has emerged as a popular parameter-efficient fine-tuning (PEFT) method for Large Language Models (LLMs), yet it still incurs notable overhead and suffers from parameter interference in multi-task scenarios. We propose LoRA with Reduced Interference (LoRI), a simple yet effective approach that freezes the projection matrices $A$ as random projections and sparsifies the matrices $B$ using task-specific masks. This design substantially reduces the number of trainable parameters while maintaining strong task performance. Moreover, LoRI minimizes cross-task interference in adapter merging by leveraging the orthogonality between adapter subspaces, and supports continual learning by using sparsity to mitigate catastrophic forgetting. Extensive experiments across natural language understanding, mathematical reasoning, code generation, and safety alignment tasks demonstrate that LoRI outperforms full fine-tuning and existing PEFT methods, while using up to 95% fewer trainable parameters than LoRA. In multi-task experiments, LoRI enables effective adapter merging and continual learning with reduced cross-task interference. Code is available at: https://github.com/juzhengz/LoRI
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。