arXiv:2602.01990cs.LGcs.AI2026-02中稿 · ICML被引 8

提出SAME模型,解决多模态持续学习中专家路由与功能漂移问题。

SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning

  • 通过正交分解稳定专家路由方向,防止任务间选择混乱。
  • 利用历史输入协方差动态调节更新,避免专家功能被覆盖。
  • 适合需要长期扩展能力的多模态大模型部署场景。

多模态大语言模型(MLLMs)通过指令微调实现强大性能,但实际部署需持续拓展能力,因此多模态持续指令微调(MCIT)至关重要。现有方法采用稀疏专家路由促进任务专业化,但我们发现随着数据分布变化,专家路由会出现漂移:例如,原本激活定位专家的视觉定位查询,在学习完OCR任务后可能被错误路由至无关专家。同时,与定位相关的专家可能被新任务覆盖而丧失原有功能。这反映出两个核心问题:路由器漂移(路由选择不一致)和专家漂移(共享专家被覆盖)。为此,我们提出稳定混合专家框架(SAME)用于MCIT。为缓解路由器漂移,SAME将路由动态分解为正交子空间,仅更新与任务相关方向;为抑制专家漂移,采用基于曲率感知的缩放策略,利用历史输入协方差在无需回放的情况下调节更新强度。此外,引入自适应专家激活机制,在训练中冻结已选专家,减少冗余计算与跨任务干扰。我们还构建了一个包含长任务序列的新基准,大量实验表明SAME达到当前最优性能。代码已开源。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually expand their capabilities, making Multimodal Continual Instruction Tuning (MCIT) essential. Recent methods leverage sparse expert routing to promote task specialization, but we find that the expert routing process suffers from drift as the data distribution evolves. For example, a grounding query that previously activated localization experts may instead be routed to irrelevant experts after learning OCR tasks. Meanwhile, the grounding-related experts can be overwritten by new tasks and lose their original functionality. Such failure reflects two problems: router drift, where expert selection becomes inconsistent over time, and expert drift, where shared experts are overwritten across tasks. Therefore, we propose StAbilized Mixture-of-Experts (SAME) for MCIT. To address router drift, SAME stabilizes expert selection by decomposing routing dynamics into orthogonal subspaces and updating only task-relevant directions. To mitigate expert drift, we regulate expert updates via curvature-aware scaling using historical input covariance in a rehearsal-free manner. SAME also introduces adaptive expert activation to freeze selected experts during training, reducing redundant computation and cross-task interference. We also introduce a new benchmark to evaluate MCIT with long task sequence, and extensive experiments demonstrate SAME's SOTA performance. Code is available at https://github.com/LAMDA-CL/Prism.

多模态持续学习专家混合Mixture-of-Experts

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。