解决多模态缺失时模型推理受限的问题,让模型能‘想象’缺失模态的信息。
AOEPT: Breaking the Implicit Modality-Reduction Bottleneck in Modality-Missing Prompt Tuning

- 用全局先验构建可复用的模态提示,替代仅依赖可见模态的旧方法
- 在多个基准上显著提升缺失模态场景下的性能,计算开销极低
- 适合需要处理不完整多模态数据的实际应用,如医疗影像分析
将多模态系统部署于真实环境时常面临模态缺失问题,即一个或多个模态不可用。尽管现有研究通过提示调优应对通用多模态变换器(MT)架构中的该挑战,但我们发现其存在根本缺陷:隐式模态缩减瓶颈。由于提示仅基于可观测模态,导致模型推理范围被限制在模态缩减子空间,无法触及缺失模态的潜在信息源。为此,我们提出AOEPT,首次引入一种新型模态上下文提示方式。具体而言,我们设计轻量级模态上下文提示(MCPs),从训练数据中提炼全局模态先验,作为缺失模态信息源的潜在存储库。在剩余模态基础上,将这些MCPs实例化为针对样本的提示,选择性补充缺失模态信息,从而恢复模型超出仅观测模态子空间的推理能力。跨多种多模态基准与骨干网络的实验验证了AOEPT的优异表现,且计算开销极小。
原文摘要 · Abstract (English)
Deploying multimodal systems in real-world environments often entails handling modality-missing scenarios, where one or more modalities are unavailable. While recent studies address this challenge for the general Multimodal Transformer (MT) architecture via prompt tuning, we identify a fundamental limitation in these methods: the Implicit Modality-Reduction bottleneck. By conditioning prompts solely on the observed modalities, they inadvertently restrict the reasoning scope of MTs to the modality-reduced subspace, cutting off access to the latent information sources of the missing modalities. To overcome this limitation, we propose AOEPT, which pioneers a novel modal-contextualized prompting fashion. Specifically, we introduce lightweight Modal-Contextualized Prompts (MCPs) that distill global modality-wise priors from training data, serving as latent repositories of the information sources for missing modalities. Conditioned on the remaining modalities, these MCPs are instantiated into instance-aware prompts that selectively augment missing-modality information for each sample, thereby restoring the reasoning scope of MTs beyond the observed-modality-only subspace. Experiments across various multimodal benchmarks and backbones confirm the strong performance of AOEPT, with minimal computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。