用微分方程控制模型合并过程,避免知识遗忘。
Unlocking the Potential of Continual Model Merging: An ODE Perspective

- 将连续模型合并视为参数空间中的轨迹演化,而非单步更新。
- 通过小样本反馈抑制性能下降方向,保留历史知识同时逼近新任务。
- 支持异质任务流,适合长期持续学习场景的模型定制。
持续模型合并(CMM)通过顺序融合任务适配模型,实现基础模型的快速定制,而无需重复训练。然而,现有合并规则通常采用固定代数或投影操作更新部署模型,难以控制历史知识与新任务知识的平衡,导致长任务流中性能不稳定甚至退化,尤其在任务效用异质时更为明显。本文提出基于常微分方程的合并框架(ODE-M),将每次合并建模为参数空间中的轨迹演化过程。受模式连通性启发,ODE-M利用修正的时间依赖速度场构建具有屏障感知的轨迹,其中轻量级一阶反馈来自小规模校准集,可抑制导致损失上升的运动,同时保留向新模型推进的进展。最终合并模型通过效用感知的时间调度选择轨迹上的运行点,实现历史知识与新任务专长之间的显式权衡。在标准CMM基准上,ODE-M在不同CLIP ViT主干网络、任务流长度和异质任务效用设置下,均持续优于强基线方法。
原文摘要 · Abstract (English)
Continual Model Merging (CMM) enables rapid customization of foundation models by sequentially incorporating task-adapted models without repeated retraining. However, existing merging rules usually update the deployed model through fixed algebraic or projection-based operations, providing limited control over how much previously accumulated knowledge should be retained relative to the incoming task model. This limitation leads to unstable retention and performance degradation in long task streams, and becomes more pronounced when tasks have heterogeneous utilities. We propose ODE-driven Merging (ODE-M), a controllable framework that formulates each continual merge as a trajectory in parameter space rather than a one-step endpoint update. Motivated by mode connectivity, ODE-M constructs a barrier-aware trajectory using a rectified time-dependent velocity field, where lightweight first-order feedback from a small calibration set suppresses loss-increasing motion while preserving progress toward the incoming model. The next merged model is then obtained by selecting an operating point along this trajectory through a utility-aware time schedule, providing an explicit mechanism for balancing retained historical knowledge and incoming task expertise. Extensive experiments on standard CMM benchmarks show that ODE-M consistently improves over strong continual merging baselines across CLIP ViT backbones, stream lengths, and heterogeneous task-utility settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。