提出双动量专家混合模型，让多模态大模型持续学习新视觉问答知识而不遗忘旧知识。
CL-MoE: Enhancing Multimodal Large Language Model with Dual Momentum Mixture-of-Experts for Continual Visual Question Answering
- 用任务级和实例级双路由选择全局与局部专家，精准分配计算资源。
- 动态动量机制根据专家与任务关系更新参数，在新旧知识间平衡。
- 在10个视觉问答任务上达领先性能，适合需要持续学习的多模态应用。
多模态大语言模型（MLLMs）在视觉语言任务（如视觉问答）中表现出色，但现实世界知识更新迅速，离线训练成本高，且面对非平稳数据流时易出现灾难性遗忘。本文提出基于持续学习的双动量专家混合框架（CL-MoE），融合MLLM与持续学习，利用大语言模型中的丰富常识知识。设计双路由专家混合（RMoE）策略，通过任务级与实例级路由选择最适配的全局和局部专家，实现稳健的专家权重分配。进一步提出动态动量专家混合（MMoE），依据专家与任务/实例间的关联动态更新参数，使模型既能吸收新知识又保留旧知识。大量实验表明，该方法在10个视觉问答任务上达到当前最优性能，验证了其有效性。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have garnered widespread attention from researchers due to their remarkable understanding and generation capabilities in visual language tasks (e.g., visual question answering). However, the rapid pace of knowledge updates in the real world makes offline training of MLLMs costly, and when faced with non-stationary data streams, MLLMs suffer from catastrophic forgetting during learning. In this paper, we propose an MLLMs-based dual momentum Mixture-of-Experts (CL-MoE) framework for continual visual question answering (VQA). We integrate MLLMs with continual learning to utilize the rich commonsense knowledge in LLMs. We introduce a Dual-Router MoE (RMoE) strategy to select the global and local experts using task-level and instance-level routers, to robustly assign weights to the experts most appropriate for the task. Then, we design a dynamic Momentum MoE (MMoE) to update the parameters of experts dynamically based on the relationships between the experts and tasks/instances, so that the model can absorb new knowledge while maintaining existing knowledge. The extensive experimental results indicate that our method achieves state-of-the-art performance on 10 VQA tasks, proving the effectiveness of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。