解决持续学习中模型合并的存储依赖问题,提升融合后模型的优化能力。
Revitalizing the Beginning: Avoiding Storage Dependency for Model Merging in Continual Learning

- 将合并过程重构为轨迹空间优化,避免传统方法的误差累积。
- 在多个基准上实现当前最优性能,显著改善新任务起始时的模型状态。
- 适合需要长期知识积累与高效模型更新的研究场景。
模型合并为将多任务专长整合进统一模型提供了有效路径,契合持续学习中知识的顺序获取。然而,保存多样化历史知识的需求与持续学习固有的存储限制存在冲突。本文系统分析了现有模型合并方法在持续学习约束下的表现,发现当前方法过度关注全局对齐,常导致任务特异性误差在连续数据流中累积放大;且后续任务初期梯度消失频繁引发优化停滞,使合并模型在下一训练阶段处于次优状态。为此,我们提出轨迹正则化合并(TRM)框架,将合并阶段重构为增强轨迹子空间中的优化过程。该框架融合任务对齐、预测一致性与梯度响应性三项协同目标,同时保持历史稳定性并重激活优化动态。大量实验表明,该方法在多个基准上达到当前最优性能。
原文摘要 · Abstract (English)
Model merging provides a compelling paradigm for integrating specialized expertise into a unified multi-task model, a goal that aligns naturally with the sequential knowledge acquisition in continual learning (CL). However, the requirement for preserving diverse forms of previous knowledge conflicts with the storage limitations inherent to CL. In this paper, we systematically analyze existing model merging methods under the constraints of CL. We find that current methods prioritize global alignment, which often leads to the accumulation and amplification of task-specific errors within the continuous data stream; and the vanishing gradients at the onset of subsequent tasks frequently cause optimization to stagnate. These leave the merged model in a suboptimal state at the beginning of the next training phase. To address these challenges, we propose Trajectory Regularized Merging (TRM), a framework that reformulates the merging phase as an optimization process within an augmented trajectory subspace. Our framework integrates three synergistic objectives including task alignment, prediction consistency, and gradient responsiveness to concurrently preserve merged model's historical stability and re-activate optimization dynamics. Extensive experimental results demonstrate that our method achieves state-of-the-art performance across multiple benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。