arXiv:2605.10765cs.CVcs.AI2026-05被引 2

为多模态持续学习设计动态生成提示,避免遗忘并提升适应性。

Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning

论文配图:Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning
图 1 · 摘自论文原文
  • 根据图文内容实时生成个性化提示,而非复用固定模板
  • 在多个基准上超越现有提示与LoRA方法,性能领先
  • 无需任务标签即可自动选择生成器,适合实际部署

多模态大语言模型通过指令微调获得强大性能,但在实际应用中常需顺序扩展新能力。为此,多模态持续指令微调(MCIT)旨在增量学习新任务的同时抑制灾难性遗忘。现有方法多采用模块组合范式:维护任务级提示或LoRA专家,在推理时动态路由或聚合部分模块。然而,同一任务内的样本在视觉场景、问题意图和推理需求上仍存在显著差异。这促使我们转向实例级适应,即针对每个查询-图像对进行个体化调整,而非仅选择或组合任务级模块。为此,本文提出DRAPE(动态跨模态提示生成),一种提示学习框架,可为MCIT生成连续的、实例特定的软提示。DRAPE不从固定池中选取提示,而是从文本指令中提取提示查询,并与视觉块特征进行交叉注意力,生成依赖于查询-图像的条件提示,再拼接到冻结的LLM前。为缓解序列更新中的遗忘,DRAPE对共享投影器应用零空间梯度投影,并在推理时采用基于CLIP的原型路由实现无任务标签的生成器选择。在多个MCIT基准上的大量实验表明,DRAPE在代表性提示与LoRA基线中达到最优性能。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, yet real-world deployment often requires continual capability expansion across sequential tasks. In such scenarios, Multimodal Continual Instruction Tuning (MCIT) aims to acquire new capabilities while limiting catastrophic forgetting. Existing methods mainly follow a module-composition paradigm: they maintain task-level prompts or LoRA experts and dynamically route or aggregate a subset of them at inference. However, samples within the same task can still differ substantially in visual scenes, question intents, and reasoning demands. This motivates instance-level adaptation to individual query-image pairs rather than only selecting or combining task-level modules. To this end, we propose DRAPE (Dynamic Cross-Modal Prompt Generation), a prompt-learning framework that synthesizes continuous instance-specific soft prompts for MCIT. Instead of selecting prompts from a fixed pool, DRAPE derives prompt queries from the textual instruction and cross-attends to visual patch features, producing query-image conditioned prompts that are prepended to the frozen LLM. To mitigate forgetting during sequential updates, DRAPE applies null-space gradient projection to the shared projector and uses CLIP-based prototype routing for task-label-free generator selection at inference. Extensive experiments on MCIT benchmarks show that DRAPE achieves state-of-the-art performance among representative prompt-based and LoRA-based continual-learning baselines.

多模态持续学习提示生成视觉语言

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。