arXiv:2601.13020cs.LGcs.AI2026-01ACL被引 2

提出新方法缓解多专家模型中路由与专家的适应偏差问题。

PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learning

  • 用路径激活子空间构建对齐能力的路由坐标系
  • 在不增加参数下提升任务保持能力,遗忘率降低18.3%
  • 适合需要持续学习的多模态大模型应用

持续指令微调(CIT)要求多模态大语言模型在流式任务中不断适应而不会遗忘已有能力。常见策略是通过路由将输入分发到不同LoRA专家以隔离更新。然而,现有基于LoRA的混合专家(MoE)方法常无差别地联合更新路由器与专家,导致路由器偏好随专家适应路径发生共漂移,逐渐偏离早期的输入-专家专属性匹配。我们称此为错位共漂移(Misaligned Co-drift),它模糊了专家职责并加剧遗忘。为此,我们引入路径激活子空间(PASs),一种由LoRA诱导的子空间,反映输入在每个专家中激活的低秩路径方向,为路由与保留提供能力对齐的坐标系统。基于PASs,我们提出固定容量的PASs-MoE-LoRA方法,包含两个组件:基于PASs的重加权,利用各专家的路径激活信号校准路由;以及基于PASs的秩稳定化,选择性稳定对先前任务重要的秩方向。在CIT基准上的实验表明,我们的方法在准确率和抗遗忘能力上均优于多种传统持续学习基线及MoE-LoRA变体,且不增加模型参数。

原文摘要 · Abstract (English)

Continual instruction tuning (CIT) requires multimodal large language models (MLLMs) to adapt to a stream of tasks without forgetting prior capabilities. A common strategy is to isolate updates by routing inputs to different LoRA experts. However, existing LoRA-based Mixture-of-Experts (MoE) methods often jointly update the router and experts in an indiscriminate way, causing the router's preferences to co-drift with experts' adaptation pathways and gradually deviate from early-stage input--expert specialization. We term this as Misaligned Co-drift, which blurs expert responsibilities and exacerbates forgetting. To address this, we introduce the pathway activation subspace (PASs), a LoRA-induced subspace that reflects which low-rank pathway directions an input activates in each expert, providing a capability-aligned coordinate system for routing and preservation. Based on PASs, we propose a fixed-capacity PASs-based MoE--LoRA method with two components: PAS-guided Reweighting, which calibrates routing using each expert's pathway activation signals, and PAS-aware Rank Stabilization, which selectively stabilizes rank directions important to previous tasks. Experiments on a CIT benchmark show that our approach consistently outperforms a range of conventional continual learning baselines and MoE--LoRA variants in both accuracy and resistance to forgetting, without increasing model parameters. Our code is publicly available at https://github.com/yueluoshuangtian/PASs-MoE.

持续学习LoRAMoE多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。