让视觉语言模型在无图时仍能准确可信地回答问题。
Bridging the Missing-Modality Gap: Improving Text-Only Calibration of Vision Language Models
- 用文本生成虚拟视觉特征，补全缺失模态信息。
- 在多个文本任务中提升准确率并降低置信度偏差。
- 适合部署在图像不可用场景的视觉语言模型优化。
视觉语言模型（VLM）常被用于仅含文本的输入，尽管其训练包含图像。我们发现移除视觉模态会导致准确率大幅下降和严重误校准，模型在纯文本提示下表现得不像原始语言主干。这一问题不仅源于语义信息缺失；即使文本描述保留关键内容，置信度仍不可靠。通过生成图像引入视觉信号可部分恢复准确性和校准性。为此，我们提出轻量级跨注意力模块LIM，从文本输入预测想象中的潜在嵌入，并注入冻结的VLM主干，无需像素级图像生成。在多个纯文本基准、未见任务及缺图场景中，LIM均提升了准确率并减少了校准误差。结果表明，潜在模态补全是实现缺失模态下可靠VLM推理的有效方法。
原文摘要 · Abstract (English)
Vision-language models (VLMs) are often deployed on text-only inputs, although they are trained with images. We find that removing the vision modality causes large drops in accuracy and severe miscalibration, and the model does not behave like its original language backbone under text-only prompting. This failure is not explained only by missing semantic information. Even when text descriptions preserve key content, confidence becomes unreliable, while adding a visual signal through generated images partially restores accuracy and calibration. We propose the Latent Imagination Module (LIM), a lightweight cross-attention module that predicts imagined latent embeddings from textual input and feeds them into a frozen VLM backbone without pixel-level image synthesis. Across text-only benchmarks, unseen tasks, and missing-image scenarios, LIM improves accuracy and reduces calibration error. These results suggest that latent modality completion is a practical approach for reliable VLM inference under missing-modality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。