用双模态提示缓解多模态持续学习遗忘，效率提升1.42倍
ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt
- 设计双模态引导的提示融合与选择机制
- 在基准上提升14.26%性能，推理速度提升1.42倍
- 适合需要高效持续学习的生成任务场景
大型多模态模型（LMMs）通过学习混合指令数据集展现出强大多任务能力。然而，在动态世界中，新任务会依次出现，迫切需要赋予LMMs多模态持续指令学习（MCIT）能力，尤其针对多样且具有挑战性的生成任务。现有MCIT方法未能充分利用LMM的独特属性，常以牺牲效率为代价获得性能。本文提出一种新型提示学习框架用于MCIT，通过自然图像-文本监督有效缓解先前知识遗忘，并管理计算复杂度。具体而言，为每项任务学习提示，利用高效的提示融合实现知识迁移，通过提示选择机制控制复杂度，均基于双模态引导。大量实验表明，该方法在MCIT基准上实现+14.26%性能提升，推理速度提升×1.42，且无计算增长。代码已公开于https://github.com/AuroraZengfh/ModalPrompt。
原文摘要 · Abstract (English)
Large Multimodal Models (LMMs) exhibit remarkable multi-tasking ability by learning mixed instruction datasets. However, novel tasks would be encountered sequentially in dynamic world, which urges for equipping LMMs with multimodal continual instruction learning (MCIT) ability especially for diverse and challenging generative tasks. Existing MCIT methods do not fully exploit the unique attribute of LMMs and often gain performance at the expense of efficiency. In this paper, we propose a novel prompt learning framework for MCIT to effectively alleviate forgetting of previous knowledge while managing computational complexity with natural image-text supervision. Concretely, we learn prompts for each task and exploit efficient prompt fusion for knowledge transfer and prompt selection for complexity management with dual-modality guidance. Extensive experiments demonstrate that our approach achieves substantial +14.26% performance gain on MCIT benchmarks with remarkable $\times$ 1.42 inference speed free from growing computation. Code is available at https://github.com/AuroraZengfh/ModalPrompt.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。