通过关键子空间投影与分级缩放，提升多任务模型融合效果。
Model Merging in the Essential Subspace
- 基于特征变化的主成分分析，提取影响表示的关键子空间。
- 在关键子空间中低秩分解并融合参数，减少任务干扰。
- 分级极化缩放增强关键知识，适合多任务模型集成场景。
模型融合旨在将多个从同一预训练检查点微调得到的任务专用模型合并为一个无需额外训练的多任务模型。尽管研究广泛，任务间干扰仍是主要障碍，常导致融合模型性能下降。本文提出一种名为ESM（Essential Subspace Merging）的鲁棒融合框架。首先对参数更新引起的特征变化进行主成分分析（PCA），所得主方向构成主导特征表示的关键子空间。每个任务的参数更新矩阵被投影到其对应的关键子空间中进行低秩分解后合并，从而缓解任务间干扰并保留核心任务特性。此外，引入多级极化缩放策略，强化含关键知识的参数，抑制冗余部分，防止重要信息在融合中被淹没。在多种任务组合与模型规模上的大量实验表明，该方法在多任务模型融合中达到当前最优性能。代码已公开于 https://github.com/kiddo127/ESM。
原文摘要 · Abstract (English)
Model merging aims to integrate multiple task-specific fine-tuned models derived from a shared pre-trained checkpoint into a single multi-task model without additional training. Despite extensive research, task interference remains a major obstacle that often undermines the performance of merged models. In this paper, we propose ESM (Essential Subspace Merging) , a robust framework for effective model merging. We begin by performing Principal Component Analysis (PCA) on feature shifts induced by parameter updates. The resulting principal directions span an essential subspace that dominantly influences feature representations. Each task's parameter update matrix is projected onto its respective essential subspace for low-rank decomposition before merging. This methodology mitigates inter-task interference while preserving core task-specific functionality. Furthermore, we introduce a multi-level polarized scaling strategy that amplifies parameters containing critical knowledge and suppresses redundant ones, preventing essential knowledge from being overwhelmed during fusion. Extensive experiments across multiple task sets and model scales demonstrate that our method achieves state-of-the-art performance in multi-task model merging. Code is available at https://github.com/kiddo127/ESM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。