通过中心化任务向量与低秩近似，显著提升模型合并性能。
Revisiting Weight Averaging for Model Merging
- 将任务向量中心化后做低秩近似，减少参数干扰。
- 在视觉任务上跨任务数量与模型规模表现稳健，准确率提升明显。
- 方法通用性强，自然语言处理任务也适用，效果媲美主流方法。
模型合并旨在不进行额外训练的情况下，通过整合独立微调模型的参数构建多任务学习器。直接对参数进行平均常因任务间参数干扰导致性能不佳。本文发现，权重平均隐式诱导出以平均值为中心的任务向量，对这些中心化向量施加低秩近似能显著提升合并效果。分析表明，中心化有效降低任务干扰，大部分任务特异性知识集中于前几个主奇异向量。该方法在视觉基准测试中表现出鲁棒且可扩展的性能，适用于不同任务数和模型规模。此外，该方法在自然语言处理任务上亦具竞争力。
原文摘要 · Abstract (English)
Model merging aims to build a multi-task learner by combining the parameters of individually fine-tuned models without additional training. While a straightforward approach is to average model parameters across tasks, this often results in suboptimal performance due to interference among parameters across tasks. In this paper, we present intriguing results that weight averaging implicitly induces task vectors centered around the weight averaging itself and that applying a low-rank approximation to these centered task vectors significantly improves merging performance. Our analysis shows that centering the task vectors effectively reduces task interference and most of task-specific knowledge is concentrated in the top singular vectors. Our method demonstrates robust and scalable performance on vision benchmarks across varying numbers of tasks and model sizes. Furthermore, we observe that our approach is applicable to natural language processing tasks with competitive performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。