让大模型持续学习新模态，无需重新训练
LLMs Can Evolve Continually on Modality for X-Modal Reasoning
- 用增量训练和适配器嵌套结构，仅靠单模态数据扩展新模态
- 在5种模态上实现接近顶尖模型的推理能力，训练参数减少98.73%
- 适合需要低成本拓展多模态能力的研究与应用
多模态大语言模型（MLLM）在多模态理解方面表现突出，但现有方法严重依赖大量特定模态预训练和联合调优，导致新增模态时计算开销巨大。本文提出 PathWeave 框架，具备模态路径切换与扩展能力，支持 MLLM 在 $ℤ$-模态推理下的持续进化。基于持续学习思想，该框架在预训练模型基础上采用增量训练策略，仅使用单模态数据即可扩展新模态，无需联合模态预训练。提出一种 Adapter-in-Adapter（AnA）结构，将单模态与跨模态适配器无缝集成，实现高效模态对齐与协作；并引入 MoE 门控模块连接两类适配器，增强多模态交互。构建了名为 MCL（Continual Learning of Modality）的挑战性基准，涵盖图像、视频、音频、深度图和点云五类高质量问答数据。大量实验表明，AnA 框架在持续学习中兼具学习可塑性与记忆稳定性。此外，PathWeave 性能媲美最先进 MLLM，同时训练参数量降低 98.73%。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have gained significant attention due to their impressive capabilities in multimodal understanding. However, existing methods rely heavily on extensive modal-specific pretraining and joint-modal tuning, leading to significant computational burdens when expanding to new modalities. In this paper, we propose PathWeave, a flexible and scalable framework with modal-Path sWitching and ExpAnsion abilities that enables MLLMs to continually EVolve on modalities for $\mathbb{X}$-modal reasoning. We leverage the concept of Continual Learning and develop an incremental training strategy atop pre-trained MLLMs, enabling their expansion to new modalities using uni-modal data, without executing joint-modal pretraining. In detail, a novel Adapter-in-Adapter (AnA) framework is introduced, in which uni-modal and cross-modal adapters are seamlessly integrated to facilitate efficient modality alignment and collaboration. Additionally, an MoE-based gating module is applied between two types of adapters to further enhance the multimodal interaction. To investigate the proposed method, we establish a challenging benchmark called Continual Learning of Modality (MCL), which consists of high-quality QA data from five distinct modalities: image, video, audio, depth and point cloud. Extensive experiments demonstrate the effectiveness of the proposed AnA framework on learning plasticity and memory stability during continual learning. Furthermore, PathWeave performs comparably to state-of-the-art MLLMs while concurrently reducing parameter training burdens by 98.73%. Our code locates at https://github.com/JiazuoYu/PathWeave
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。