首篇系统综述多模态持续学习,梳理方法与挑战。
Recent Advances of Multimodal Continual Learning: A Comprehensive Survey
- 按正则、架构、重放、提示四类整理多模态持续学习方法
- 揭示模态失衡、灾难性遗忘等核心难题及性能瓶颈
- 适合想入门或深耕多模态持续学习的研究者参考
持续学习旨在让模型在不断接收新数据的同时,保留已有知识而不遗忘。随着模型从小型走向大规模预训练架构,任务也从单模态扩展到多模态,多模态持续学习(MMCL)应运而生。其主要难点在于不能简单叠加单模态持续学习方法,直接拼接常导致多模态灾难性遗忘,表现不佳。此外,MMCL还带来新挑战:模态不平衡、复杂模态交互、高计算成本,以及多模态骨干网络零样本能力的退化。本文首次全面综述MMCL,提供背景知识与设定,构建系统分类体系,将方法分为正则化、架构设计、重放机制与提示学习四类,详述其原理与创新点。同时汇总开放数据集与基准,深入讨论现有问题,并展望未来方向。项目资源已托管于GitHub:https://github.com/LucyDYu/Awesome-Multimodal-Continual-Learning。
原文摘要 · Abstract (English)
Continual learning (CL) aims to empower machine learning models to learn continually from new data, while building upon previously acquired knowledge without forgetting. As models have evolved from small to large pre-trained architectures, and from supporting unimodal to multimodal data, multimodal continual learning (MMCL) methods have recently emerged. The primary complexity of MMCL is that it extends beyond a simple stacking of unimodal CL methods. Such straightforward approaches often suffer from multimodal catastrophic forgetting, yielding unsatisfactory performance. In addition, MMCL introduces new challenges that unimodal CL methods fail to adequately address, including modality imbalance, complex modality interaction, high computational costs, and degradation of pre-trained zero-shot capability of multimodal backbones. In this work, we present the first comprehensive survey on MMCL. We provide essential background knowledge and MMCL settings, as well as a structured taxonomy of MMCL methods. We categorize MMCL methods into four categories, i.e., regularization-based, architecture-based, replay-based, and prompt-based methods, explaining their methodologies and highlighting their key innovations. Additionally, to prompt further research in this field, we summarize open MMCL datasets and benchmarks, provide an in-depth discussion, and discuss several promising future directions. We have also created a GitHub repository for indexing relevant MMCL papers and open resources available at https://github.com/LucyDYu/Awesome-Multimodal-Continual-Learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。