通过奇异值分解实现多任务自适应，解决任务冲突与遗忘问题。
MoORE: SVD-based Model MoE-ization for Conflict- and Oblivion-Resistant Multi-Task Adaptation
- 基于SVD分解权重矩阵，构建正交秩一专家混合模型
- 在多个数据集上优于现有方法，有效缓解任务冲突与遗忘
- 适合需要稳定多任务学习的场景，如持续学习、迁移学习
在多任务场景中，大规模基础模型的适配常面临任务冲突与遗忘问题。为此，我们提出一种新型「模型专家化」策略，实现抗冲突与抗遗忘的多任务适配。给定预训练模型的权重矩阵，该方法先进行奇异值分解（SVD），再引入可学习路由器根据任务和样本调整奇异值，使权重矩阵变为正交秩一专家混合（MoORE），每个专家对应左右奇异向量的外积。通过在右奇异向量上施加可学习正交变换，可提升模型容量。与低秩适配（LoRA）及其专家驱动变体不同，MoORE保证专家间正交性，并维持原始权重矩阵的列空间。这两项特性分别使模型具备抵抗新任务冲突和避免原任务遗忘的能力。在多个数据集上的实验表明，MoORE在多任务适配中始终优于现有方法，验证了其抗冲突与抗遗忘优势。代码已公开于https://github.com/DaShenZi721/MoORE。
原文摘要 · Abstract (English)
Adapting large-scale foundation models in multi-task scenarios often suffers from task conflict and oblivion. To mitigate such issues, we propose a novel ''model MoE-ization'' strategy that leads to a conflict- and oblivion-resistant multi-task adaptation method. Given a weight matrix of a pre-trained model, our method applies SVD to it and introduces a learnable router to adjust its singular values based on tasks and samples. Accordingly, the weight matrix becomes a Mixture of Orthogonal Rank-one Experts (MoORE), in which each expert corresponds to the outer product of a left singular vector and the corresponding right one. We can improve the model capacity by imposing a learnable orthogonal transform on the right singular vectors. Unlike low-rank adaptation (LoRA) and its MoE-driven variants, MoORE guarantees the experts' orthogonality and maintains the column space of the original weight matrix. These two properties make the adapted model resistant to the conflicts among the new tasks and the oblivion of its original tasks, respectively. Experiments on various datasets demonstrate that MoORE outperforms existing multi-task adaptation methods consistently, showing its superiority in terms of conflict- and oblivion-resistance. The code of the experiments is available at https://github.com/DaShenZi721/MoORE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。