arXiv:2606.02502cs.CL2026-06中稿 · EMNLP

CRAM通过动态路由与自适应专家分配,实现多模态持续学习不遗忘、少参数消耗。

CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning

论文配图:CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning
图 1 · 摘自论文原文
  • 按任务特征分离模块,用中心点路由激活已有专家
  • 根据能力缺口自适应分配参数,仅更新必要部分
  • 适合长期扩展新任务的多模态模型,兼顾效率与稳定性

多模态大语言模型(MLLMs)通过指令微调统一视觉-语言任务,但真实部署需持续扩展能力,因此多模态持续指令微调(MCIT)至关重要。现有方法或共享全部参数导致任务间遗忘,或为每项任务独立分配模块,造成参数效率低下。本文提出CRAM(Centroid-Routing and Adaptive MoE),将任务特异性模式隔离至独立模块,缓解灾难性遗忘。通过自适应秩实例化识别现有专家能力与新任务需求间的差距,仅动态分配必要参数。利用中心点引导路由识别并激活已有专家能力,正交性惩罚限制新更新方向,防止重复学习通用能力。在多个基准上的实验验证其优于现有方法。代码已开源。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) unify heterogeneous vision-language tasks under a shared generative framework via instruction tuning, yet real-world deployment demands continuous capability expansion, making Multimodal Continual Instruction Tuning (MCIT) essential. Existing methods either update all tasks with a shared parameter set or allocate dedicated modules for each new task. Shared updates force heterogeneous tasks to compete, causing forgetting of learned capabilities. Conversely, isolated expansion prevents interference but severely limits parameter efficiency over long task streams. To address this dilemma, we propose CRAM (Centroid-Routing and Adaptive MoE). Specifically, by isolating task-specific patterns into independent modules, CRAM mitigates catastrophic forgetting across tasks. To further boost parameter efficiency, we utilize adaptive-rank instantiation to identify the capability gap between existing expert capability and new task demands, and dynamically allocate only the necessary parameters. To ensure stable reuse among tasks, centroid-guided routing recognizes and activates existing experts' capabilities, while an orthogonality penalty confines new updates to task-specific directions, preventing re-learning general capability. Extensive experiments across diverse benchmarks demonstrate its superiority over existing methods. Code is available at https://github.com/LAMDA-CL/EMNLP2026-CRAM.

多模态持续学习MoE参数效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。