提升多模型合并效果，避免任务信息被淹没
Subspace-Boosted Model Merging
- 通过子空间增强保持任务向量秩，防止信息衰减
- 在20个专家模型合并时，性能提升超10%
- 可量化任务相似性，适合需要高效融合的场景
模型合并可将多个专业模型融合为一个能执行多项任务的统一模型。然而，随着合并的专家模型增多，性能增益逐渐减弱，甚至下降。本文实证并理论分析发现，基于任务算术的方法中，合并越多专家，共性信息越主导，导致任务特异性信息被淹没，引发秩坍缩。为此，提出子空间增强（Subspace Boosting），在奇异值分解后的任务向量空间中操作，维持任务向量秩。该方法在视觉与语言基准上，对最多20个专家模型的合并，性能提升超过10%。此外，提出使用高阶广义奇异值分解量化任务相似性，提供模型合并的新可解释视角。代码与模型已公开于 https://github.com/ronskoro/Subspace-Boosting。
原文摘要 · Abstract (English)
Model merging enables the combination of multiple specialized expert models into a single model capable of performing multiple tasks. However, the benefits of merging an increasing amount of specialized experts generally lead to diminishing returns and reduced overall performance gains. In this work, we empirically and theoretically analyze this limitation, proving that for Task Arithmetic-based methods, as more experts are merged, the common information dominates the task-specific information, leading to inevitable rank collapse. To mitigate this issue, we introduce Subspace Boosting, which operates on the singular value decomposed task vector space and maintains task vector ranks. Subspace Boosting raises merging efficacy for up to 20 experts by large margins of more than 10% when evaluated on both vision and language benchmarks. Moreover, we propose employing Higher-Order Generalized Singular Value Decomposition to quantify task similarity, offering a new interpretable perspective on model merging. Code and models are available at https://github.com/ronskoro/Subspace-Boosting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。