arXiv:2606.19164cs.LGcs.AI2026-06

通过聚焦关键参数方向,实现多任务模型高效融合。

Essential Subspace Merging for Multi-Task Learning

论文配图:Essential Subspace Merging for Multi-Task Learning
图 1 · 摘自论文原文
  • 识别任务更新中的核心参数方向,构建关键子空间。
  • 静态融合方法减少干扰,保持各任务性能,提升模型压缩率。
  • 动态版本支持推理时按需选择专家,适合复杂多任务场景。

模型融合旨在通过整合从同一预训练检查点微调的多个模型能力,实现多任务学习。其核心挑战在于不同任务间参数更新引发的干扰。本文分析任务更新引起的输出偏移,发现其能量集中在少数主方向上,这些方向构成的子空间称为关键子空间。相比之下,其余方向虽携带少量任务相关能量,但累积后会导致融合时严重干扰。基于此,我们提出关键子空间分解(ESD),将每个任务更新按激活偏移的主成分进行分解。在此基础上,提出无需训练的静态融合方法ESM,通过正交化并融合关键成分生成紧凑的多任务模型。进一步扩展为ESM++,一种无需训练的动态融合方法,将任务特定残差分解为低秩专家,并在前向推理中通过原型路由选择最相关专家。大量实验表明,ESM与ESM++在多种任务组合和模型规模下均有效保留任务知识并降低任务间干扰。代码已开源:https://github.com/kiddo127/ESM。

原文摘要 · Abstract (English)

Model merging aims to enable multi-task learning by integrating the capabilities of multiple models fine-tuned from the same pre-trained checkpoint into a single model. Its core challenge is inter-task interference among task-specific parameter updates. In this paper, we analyze the output shifts induced by task updates and observe that their energy is concentrated in a small number of principal directions. We call the subspace spanned by these directions the essential subspace. In contrast, most remaining directions carry little task-relevant energy, but their accumulation across multiple task updates can cause severe interference during merging. Motivated by this observation, we propose Essential Subspace Decomposition (ESD), which decomposes each task update according to the principal components of its activation shift. Based on ESD, we introduce Essential Subspace Merging (ESM), a training-free static merging method that orthogonalizes and fuses essential components into one compact multi-task model. We further extend ESM to ESM++, a training-free dynamic merging method that decomposes task-specific residuals into low-rank experts and selects the most relevant expert through prototype-based routing during forward inference. Extensive experiments across multiple task sets and model scales demonstrate that ESM and ESM++ effectively preserves task knowledge while reducing inter-task interference. Code is available at https://github.com/kiddo127/ESM.

多任务学习模型融合参数压缩静态融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。