通过分析模型层间任务向量，减少模型融合中的任务干扰。
Task Singular Vectors: Reducing Task Interference in Model Merging
- 从层级视角提取任务奇异向量，捕捉关键结构信息
- 压缩至原大小10%仍保持99%准确率，显著降低冗余
- 基于奇异向量交互设计新融合方法，有效缓解任务冲突
任务算术已成为一种无需额外训练即可合并模型的简单而有效的方法。然而，该方法将整个网络视为平坦的参数向量，忽略了关键的结构信息，容易受到任务干扰。本文研究了层级层面的任务向量，重点关注任务层矩阵及其奇异值分解。特别地，我们关注由此产生的奇异向量，称之为任务奇异向量（TSV）。鉴于层任务矩阵通常为低秩，我们提出TSV-Compress（TSV-C），一种简单方法，将矩阵压缩至原始大小的10%，同时保留99%的准确率。进一步地，我们利用这一低秩空间定义了一种基于不同任务奇异向量交互的新任务干扰度量。基于这些发现，我们提出了新的模型融合方法TSV-Merge（TSV-M），结合压缩与干扰抑制，显著优于现有方法。
原文摘要 · Abstract (English)
Task Arithmetic has emerged as a simple yet effective method to merge models without additional training. However, by treating entire networks as flat parameter vectors, it overlooks key structural information and is susceptible to task interference. In this paper, we study task vectors at the layer level, focusing on task layer matrices and their singular value decomposition. In particular, we concentrate on the resulting singular vectors, which we refer to as Task Singular Vectors (TSV). Recognizing that layer task matrices are often low-rank, we propose TSV-Compress (TSV-C), a simple procedure that compresses them to 10% of their original size while retaining 99% of accuracy. We further leverage this low-rank space to define a new measure of task interference based on the interaction of singular vectors from different tasks. Building on these findings, we introduce TSV-Merge (TSV-M), a novel model merging approach that combines compression with interference reduction, significantly outperforming existing methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。