提出任务干扰统一框架，揭示优化器如何通过控制谱范数提升持续学习与模型融合性能。
When Muon Meets Task Interference: A Spectral Perspective on Continual Learning and Model Merging
- 将持续学习与模型融合的难题统一为任务干扰，用层间弗罗贝尼乌斯内积量化。
- 理论证明谱范数是干扰关键因子，实验显示替换AdamW后准确率最高提升5.02点。
- 新优化器Muon通过调控谱范数有效抑制干扰，适合需要稳定多任务学习的场景。
持续学习（CL）与模型融合（MM）均旨在获得在多个任务上表现良好的单一模型，分别受灾难性遗忘和权重解耦误差挑战。现有研究将二者独立处理，且忽略基础优化器的几何影响。本文揭示两者实为同一现象：某任务的参数更新会改变模型在另一任务上的输出。我们将其形式化为任务干扰，并简化为逐层弗罗贝尼乌斯内积⟨ΔWℓ, Jℓ(x)⟩F。该量揭示了优化器的作用：理论推导出上界，其中谱范数‖ΔWℓ‖₂为可由优化器控制的干扰因子，且每模式分析表明该上界能追踪经验干扰的主要部分。具体地，我们识别出近期提出的Muon优化器正是通过构造方式调节此因子。本工作揭示其对谱范数的优雅控制可同时收紧CL与MM的干扰上界，使Muon成为一种与现有方法互补的优化器中心型方案。理论分析经实验充分验证：在三个CLIP骨干网络上，将AdamW替换为Muon，在八任务模型融合基准上准确率最高提升+5.02点；在持续学习中，于十种类增量、三种任务增量及11任务MTIL基准上均实现一致正向增益。
原文摘要 · Abstract (English)
Continual learning (CL) and model merging (MM) both aim to obtain a single model that performs well across multiple tasks, challenged respectively by catastrophic forgetting and weight-disentanglement error. In the literature, these difficulties are merely treated separately and mitigated through a variety of solutions, while the geometry induced by the base optimizer is treated as an implementation detail. In this work, we show that the two difficulties are in fact two instances of the same phenomenon: a parameter update useful for one task shifts the model's outputs on another. We formalize this shared phenomenon as \textit{task interference} and reduce it to a common layer-wise Frobenius inner product $\langle ΔW_\ell, J_\ell(x)\rangle_F$. This quantity, in turn, is utilized to expose the role of the optimizer. We theoretically derive an upper bound that isolates the spectral norm $\|ΔW_\ell\|_2$ as an optimizer-controllable factor of task interference, and a per-mode analysis shows that this bound tracks the dominant part of the empirical interference. Specifically, we then identify the recent Muon optimizer as a mechanism that regulates this factor by construction. Our work reveals that its elegant control on spectral norm tightens the interference bound for both CL and MM, positioning Muon as a principled optimizer-centric approach complementary to existing solutions. Our theoretcal analysis is well validated by experimental results. Replacing the AdamW optimizer with Muon improves accuracy by up to +5.02 points on the eight-task model-merging benchmark across three CLIP backbones. For continual learning, Muon also delivers uniformly positive gains across ten class-incremental protocols, three task-incremental protocols, and the 11-task MTIL benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。