用稀疏编码分离任务特征,提升多任务模型融合效果。
Escaping Low-Dimensional Overlap: Multi-Task Model Merging via High-Dimensional Sparse Disentanglement

- 通过稀疏自编码器将任务向量映射到高维稀疏空间,实现特征解耦
- 在4个任务冲突场景下,相比最强基线提升2.78%性能
- 轻量级优化器只合并关键层,降低计算开销,适合资源受限场景
模型融合能高效构建无需额外训练的多任务通用模型,但严重任务干扰会导致性能下降。其主要原因是参数空间中的特征超位置重叠,使任务特异性特征相互纠缠,传统分解方法难以有效分离有用任务方向。本文提出基于稀疏表示的融合框架,利用稀疏自编码器(SAEs)将任务向量投影至高维稀疏特征空间,实现融合前的特征级解耦。为降低计算开销,进一步引入轻量级分组排名零阶优化器(GR-ZOO),识别任务关键层进行选择性融合。在Qwen2.5-1.5B和Qwen2.5-7B上的实验表明,该方法在数学推理、代码生成、指令遵循和通用知识等任务上均优于多项代表性基线,包括Task Arithmetic、TIES-Merge、DARE、Fisher-Merge及若干近期无训练融合方法。在Qwen2.5-1.5B上的高度冲突四任务设置中,性能较最强基线提升2.78%。
原文摘要 · Abstract (English)
Model merging provides an efficient way to construct multi-task generalist models without additional training, but its performance often degrades under severe task interference. Task interference in model merging primarily stems from \textit{superposition}, where task-specific features become entangled within the parameter space. This entanglement renders conventional decomposition methods insufficient for effectively isolating useful task directions from interfering components. In this paper, we propose a sparse-representation-based merging framework that uses Sparse Autoencoders (SAEs) to project task vectors into a high-dimensional sparse feature space, enabling feature-level disentanglement before fusion. To reduce computational overhead, we further introduce a lightweight Group-Ranked Zeroth-Order Optimizer (GR-ZOO) to identify task-critical layers for selective merging. Experiments on both Qwen2.5-1.5B and Qwen2.5-7B demonstrate that our method consistently outperforms representative baselines, including Task Arithmetic, TIES-Merge, DARE, Fisher-Merge,and several recent training-free merging methods, across mathematical reasoning, code generation, instruction following, and general knowledge tasks. In a highly conflicting four-task setting on Qwen2.5-1.5B, our method further achieves a 2.78\% improvement over the strongest baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。