通过统一子空间提升模型融合效果，实现无训练的多任务模型性能突破。
No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
- 基于任务矩阵奇异成分对齐，提出各向同性融合框架。
- 在视觉与语言任务上超越现有方法，性能接近单任务模型。
- 适用于不同规模模型，无需额外训练，适合实际部署场景。
模型融合将多个特定任务模型的权重整合为单一多任务模型。尽管该问题受到广泛关注，但合并模型与单任务模型之间仍存在显著性能差距。本文研究任务矩阵——即对预训练模型施加的权重更新矩阵——的关键特性，发现任务特定矩阵与融合矩阵的奇异分量对齐程度与性能提升强相关。基于此，我们提出各向同性融合框架，通过平滑任务矩阵的奇异值谱，增强对齐性并缩小性能差距。此外，引入通用与任务特异性子空间进一步优化对齐与性能。所提方法在多种任务组合与模型规模下均达到当前最优表现，深化了对模型融合机制的理解，提供了一种无需额外训练的高效融合方案。代码已开源：https://github.com/danielm1405/iso-merging。
原文摘要 · Abstract (English)
Model merging integrates the weights of multiple task-specific models into a single multi-task model. Despite recent interest in the problem, a significant performance gap between the combined and single-task models remains. In this paper, we investigate the key characteristics of task matrices -- weight update matrices applied to a pre-trained model -- that enable effective merging. We show that alignment between singular components of task-specific and merged matrices strongly correlates with performance improvement over the pre-trained model. Based on this, we propose an isotropic merging framework that flattens the singular value spectrum of task matrices, enhances alignment, and reduces the performance gap. Additionally, we incorporate both common and task-specific subspaces to further improve alignment and performance. Our proposed approach achieves state-of-the-art performance on vision and language tasks across various sets of tasks and model scales. This work advances the understanding of model merging dynamics, offering an effective methodology to merge models without requiring additional training. Code is available at https://github.com/danielm1405/iso-merging .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。