让多模态大模型持续学习新知识，减少遗忘。
MLLM-CL: Continual Learning for Multimodal Large Language Models
- 通过参数隔离和智能路由机制防止旧知识被覆盖。
- 在新领域和新能力上都能低遗忘地持续学习。
- 适合研究持续学习或实际部署多模态模型的人。
近年来的多模态大语言模型（MLLMs）在视觉-语言理解方面表现优异，但在动态现实场景中持续整合新知识与技能时面临挑战。尽管持续学习（CL）提供了潜在解决方案，但现有基准与方法存在显著局限。本文提出 MLLM-CL，一个涵盖领域持续学习与能力持续学习的新基准：前者针对随时间演进的主流领域进行独立同分布（IID）评估，后者则在非独立同分布（non-IID）场景下评估新增模型能力。方法上，我们采用参数隔离与基于MLLM的路由机制，有效缓解灾难性遗忘。大量实验表明，该方法能以极小遗忘实现领域知识与功能能力的持续集成，显著优于现有方法。相关基准与代码已开源于 https://github.com/bjzhb666/MLLM-CL。
原文摘要 · Abstract (English)
Recent Multimodal Large Language Models (MLLMs) excel in vision-language understanding but face challenges in adapting to dynamic real-world scenarios that require continuous integration of new knowledge and skills. While continual learning (CL) offers a potential solution, existing benchmarks and methods suffer from critical limitations. In this paper, we introduce MLLM-CL, a novel benchmark encompassing domain and ability continual learning, where the former focuses on independently and identically distributed (IID) evaluation across evolving mainstream domains, whereas the latter evaluates on non-IID scenarios with new model abilities. Methodologically, we propose preventing catastrophic interference through parameter isolation and an MLLM-based routing mechanism. Extensive experiments demonstrate that our approach can integrate domain-specific knowledge and functional abilities with minimal forgetting, significantly outperforming existing methods. Our benchmark and code are available at https://github.com/bjzhb666/MLLM-CL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。