通过注意力感知机制融合多个模型,提升跨领域性能。
Saliency-Aware Model Merging

- 基于结构剪枝的显著性评分,动态评估各模型贡献度。
- 在多个视觉与语言任务上,性能逼近测试时自适应方法。
- 适用于低资源场景,特别适合融合微调过的LoRA模块。
模型融合旨在将针对不同数据集微调的多个专用模型合并为统一架构,实现跨领域泛化能力。现有无数据模型融合方法因依赖简单的参数级启发式策略,忽视层间依赖关系及知识分布不均,难以扩展。本文提出SA-Merging,基于结构剪枝中的连通性显著性理论(如SynFlow),将其拓展至无数据模型融合场景。定义相对于共享基础模型的任务向量显著性分数,并引入融合感知调制,通过专家间一致性降低任务干扰。在此基础上,设计迭代显著性感知融合流程,逐步剔除无效更新,同时保持端到端连通性。此外,进一步提出秩级显著性分解方法,应用于LoRA时可保留其结构完整性。在视觉与语言任务上的大量实验表明,该显著性驱动方法显著缩小了无数据融合与测试时自适应方法间的性能差距。
原文摘要 · Abstract (English)
Model merging aims to consolidate multiple task-specific models fine-tuned on different datasets into a unified architecture that performs cross-domain proficiency. Current data-free model merging methods often struggle to scale as they rely on simple parameter-level heuristics that ignore inter-layer dependencies and non-uniform distribution of expertise. This work proposes SA-Merging, which is built upon connectivity-based saliency formulations from structural pruning (e.g., SynFlow) and extends them to the data-free model merging setting. We define a saliency score over task vectors relative to a shared base model, and further introduce merge-aware modulation that incorporates agreement across experts to mitigate task interference. Based on this formulation, an iterative saliency-aware merging procedure progressively removes non-informative updates while preserving end-to-end connectivity. Furthermore, we extend SA-Merging to introduce rank-wise saliency decomposition for LoRAs without compromising their structural integrity. Extensive experiments on vision and language tasks demonstrate the effectiveness of our saliency-based approach, further reducing the gap between data-free and test-time adaptation methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。