arXiv:2606.26902cs.AI2026-06被引 1

从合并模型中恢复任务专家,提升多任务泛化能力。

Learning to Recover Task Experts from a Multi-Task Merged Model

论文配图:Learning to Recover Task Experts from a Multi-Task Merged Model
图 1 · 摘自论文原文
  • 通过预测参数偏移量,还原被干扰的专家权重。
  • 在视觉与NLP任务中恢复超95%单专家性能。
  • 无需路由器,基于SVD子空间签名识别任务类型。

多任务模型合并旨在将多个任务专用专家整合为统一模型,但静态合并常因参数干扰导致性能下降。动态合并虽可缓解此问题,却需在推理时加载冗余专家组件,成本高昂。本文从任务专家视角出发,将参数干扰视为合并过程中引入的参数扰动,并证明其可建模为仿射变换,近似为加性偏移。为此提出ReTeX框架,通过预测偏移量来消除参数干扰,恢复任务专家性能。为在任务身份未知时选择合适专家,设计无需路由器的离线SVD子空间签名标识器,依据输入投影残差最小者选择任务。实验表明,ReTeX在视觉与NLP领域均恢复超过95%的独立专家性能,并显著提升对未见任务的泛化能力。更重要的是,参数偏移预测催生出对分布外任务的自适应知识插值能力,使模型能融合已有专家知识处理新任务。代码已开源。

原文摘要 · Abstract (English)

Multi-task model merging aims to consolidate several task-specific experts into a unified model, yet static merging consistently suffers from parameter interference. While dynamic merging models aim to bridge this gap, many works rely on the costly storage and loading of redundant expert components at inference. In this work, from the perspective of task expert, we view parameter interference as parameter perturbation introduced to each expert during merging process. We show that such parameter perturbations can be modeled as affine transformation, which can be approximated as additive offsets. Motivated by these, we propose Recover Task eXpert (ReTeX), a framework that predicts those offsets, in order to undo parameter interference and recover task-expert performance from a single merged checkpoint. To recover the appropriate expert when task identity is unknown, we introduce a router-free task identifier based on SVD subspace signatures computed offline before inference. At inference, the identifier selects the task whose subspace yields the smallest projection residual for a given input. As a result, ReTeX recovers over 95% of individual-expert performance in both vision and NLP domains, while significantly improving generalization to unseen tasks. Crucially, we also show that the parameter offset prediction leads to emergent adaptive interpolation of expert knowledge for out-of-distribution (OOD) tasks. ReTeX adaptively interpolates seen expert knowledge to handle unseen tasks. Our code is available at https://github.com/BAIKLAB/ReTeX

多任务学习模型合并专家恢复自适应插值

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。