提出解耦原型学习，让视觉语言模型在缺图或缺文时仍能准确预测。
DPL: Decoupled Prototype Learning for Enhancing Robustness of Vision-Language Transformers to Missing Modalities
- 为每类样本设计适配缺失模态的专属原型
- 原型分解为图像与文本特有成分，动态调整决策路径
- 兼容现有提示框架，提升多模态模型鲁棒性
视觉语言模型在输入模态（如图像）缺失时性能急剧下降，因模型需基于不完整信息进行预测。现有感知缺失的提示方法虽减轻了性能损失，但仍依赖传统分类头（如全连接层），其计算方式不随模态缺失情况变化。本文提出解耦原型学习（DPL），一种新型分类头架构，能根据实际观测到的模态动态调整决策过程。对每个类别，DPL根据当前缺失情况（图像缺失、文本缺失或混合缺失）选择特定原型，并将每个原型分解为图像特异和文本特异成分，使分类决策真正依赖于可用信息。该自适应设计显著提升模型在模态缺失场景下的表现，且完全兼容现有基于提示的框架。在MM-IMDb、UPMC Food-101和Hateful Memes上的大量实验表明，DPL在所有主流多模态图文数据集及多种缺失情形下均优于现有最先进方法。
原文摘要 · Abstract (English)
The performance of Visio-Language Transformers drops sharply when an input modality (e.g., image) is missing, because the model is forced to make predictions using incomplete information. Existing missing-aware prompt methods help reduce this degradation, but they still rely on conventional prediction heads (e.g., a Fully-Connected layer) that compute class scores in the same way regardless of which modality is present or absent. We introduce Decoupled Prototype Learning (DPL), a new prediction head architecture that explicitly adjusts its decision process to the observed input modalities. For each class, DPL selects a set of prototypes specific to the current missing-modality cases (image-missing, text-missing, or mixed-missing). Each prototype is then decomposed into image-specific and text-specific components, enabling the head to make decisions that depend on the information actually present. This adaptive design allows DPL to handle inputs with missing modalities more effectively while remaining fully compatible with existing prompt-based frameworks. Extensive experiments on MM-IMDb, UPMC Food-101, and Hateful Memes demonstrate that DPL outperforms state-of-the-art approaches across all widely used multimodal imag-text datasets and various missing cases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。