arXiv:2511.19561cs.LGcs.AI2025-11

用最优传输方法融合任务模型,避免遗忘且高效扩展。

Merging without Forgetting: Continual Fusion of Task-Specific Models via Optimal Transport

  • 基于最优传输理论寻找共享掩码,对齐不同任务的语义结构。
  • 在多个视觉与语言任务上超越现有方法,准确率与效率双优。
  • 支持持续融合新任务,内存占用固定,适合实际多任务系统。

将针对不同任务微调的模型合并为单一统一模型,已成为构建高效多任务系统的重要方向。现有方法主要依赖权重空间的参数插值,但我们发现这会在特征空间引入显著分布偏移,损害任务特异性知识。本文提出基于最优传输的掩码融合(OTMF)框架,通过最优传输计划发现适用于任务向量的共同掩码,对齐各任务模型的语义几何结构。该方法不直接聚合特征或权重,而是选择性提取可迁移的通用组件,同时保留每个任务的独特结构身份。为确保实际应用中的可扩展性,OTMF进一步支持持续融合范式,可增量整合新任务向量而无需回溯先前任务,保持有限内存占用,实现对不断增长任务集的高效融合。我们在多个视觉与语言基准上进行实验,结果表明OTMF在准确率和效率方面均达到当前最优水平。这些发现凸显了本方法在模型融合中的实用与理论价值。

原文摘要 · Abstract (English)

Merging models fine-tuned for different tasks into a single unified model has become an increasingly important direction for building versatile, efficient multi-task systems. Existing approaches predominantly rely on parameter interpolation in weight space, which we show introduces significant distribution shift in the feature space and undermines task-specific knowledge. In this paper, we propose OTMF (Optimal Transport-based Masked Fusion), a novel model merging framework rooted in optimal transport theory to address the distribution shift that arises from naive parameter interpolation. Instead of directly aggregating features or weights, OTMF aligns the semantic geometry of task-specific models by discovering common masks applied to task vectors through optimal transport plans. These masks selectively extract transferable and task-agnostic components while preserving the unique structural identities of each task. To ensure scalability in real-world settings, OTMF further supports a continual fusion paradigm that incrementally integrates each new task vector without revisiting previous ones, maintaining a bounded memory footprint and enabling efficient fusion across a growing number of tasks. We conduct comprehensive experiments on multiple vision and language benchmarks, and results show that OTMF achieves state-of-the-art performance in terms of both accuracy and efficiency. These findings highlight the practical and theoretical value of our approach to model merging.

模型融合最优传输持续学习多任务

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。