用检索增强动态提示提升缺失模态下的多模态学习性能
Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learning
- 通过检索相似样本生成动态提示，适应不同缺失情况
- 在三个真实数据集上均优于现有方法，显著提升模型鲁棒性
- 适合处理模态缺失场景的多模态模型研究者使用
不完整模态的多模态学习具有实际意义但挑战重重。近期研究通过可学习提示增强预训练多模态变换器（MMTs）在模态缺失时的鲁棒性，但存在三方面局限：(1) 缺失模态提供的线索有限，影响任务推理；(2) 虚拟填充导致信息丢失并引入噪声；(3) 静态提示与实例无关，难以应对多样缺失情形。为此，我们提出RAGPT——一种检索增强的动态提示调优框架。该框架包含三个模块：(I) 多通道检索器，通过模内检索定位相似实例；(II) 缺失模态生成器，利用检索上下文恢复缺失内容；(III) 上下文感知提示器，从相关实例中捕捉上下文知识并生成动态提示，显著增强MMT鲁棒性。在三个真实数据集上的大量实验表明，RAGPT持续优于所有对比基线。
原文摘要 · Abstract (English)
Multimodal learning with incomplete modality is practical and challenging. Recently, researchers have focused on enhancing the robustness of pre-trained MultiModal Transformers (MMTs) under missing modality conditions by applying learnable prompts. However, these prompt-based methods face several limitations: (1) incomplete modalities provide restricted modal cues for task-specific inference, (2) dummy imputation for missing content causes information loss and introduces noise, and (3) static prompts are instance-agnostic, offering limited knowledge for instances with various missing conditions. To address these issues, we propose RAGPT, a novel Retrieval-AuGmented dynamic Prompt Tuning framework. RAGPT comprises three modules: (I) the multi-channel retriever, which identifies similar instances through a within-modality retrieval strategy, (II) the missing modality generator, which recovers missing information using retrieved contexts, and (III) the context-aware prompter, which captures contextual knowledge from relevant instances and generates dynamic prompts to largely enhance the MMT's robustness. Extensive experiments conducted on three real-world datasets show that RAGPT consistently outperforms all competitive baselines in handling incomplete modality problems. The code of our work and prompt-based baselines is available at https://github.com/Jian-Lang/RAGPT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。