给冻结的视觉语言模型加不确定度，提升推理可靠性。
Probabilistic Embeddings for Frozen Vision-Language Models: Uncertainty Quantification with Gaussian Process Latent Variable Models
- 用高斯过程学习图像文本共享低维空间的随机表示
- 在跨模态检索等任务中实现最优不确定性校准
- 无需重新训练，可直接用于CLIP等预训练模型
视觉语言模型（VLM）通过将图像和文本映射到共享隐空间来学习联合表示。然而，现有研究指出，标准VLM的确定性嵌入难以捕捉视觉与文本描述中的模糊性及图像与文本间多重对应关系带来的不确定性。现有方法虽尝试在训练时学习概率嵌入，但需大量数据，且无法利用CLIP等大规模VLM已学得的强大表征。本文提出GroVE，一种针对冻结VLM的后处理概率嵌入方法。GroVE基于高斯过程潜在变量模型（GPLVM），学习一个共享的低维隐空间，通过单模态重建与跨模态对齐目标优化图像与文本输入的统一表示。训练完成后，高斯过程模型生成具有不确定性感知的概率嵌入。实验表明，GroVE在跨模态检索、视觉问答与主动学习等多个下游任务中实现了最先进的不确定性校准效果。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) learn joint representations by mapping images and text into a shared latent space. However, recent research highlights that deterministic embeddings from standard VLMs often struggle to capture the uncertainties arising from the ambiguities in visual and textual descriptions and the multiple possible correspondences between images and texts. Existing approaches tackle this by learning probabilistic embeddings during VLM training, which demands large datasets and does not leverage the powerful representations already learned by large-scale VLMs like CLIP. In this paper, we propose GroVE, a post-hoc approach to obtaining probabilistic embeddings from frozen VLMs. GroVE builds on Gaussian Process Latent Variable Model (GPLVM) to learn a shared low-dimensional latent space where image and text inputs are mapped to a unified representation, optimized through single-modal embedding reconstruction and cross-modal alignment objectives. Once trained, the Gaussian Process model generates uncertainty-aware probabilistic embeddings. Evaluation shows that GroVE achieves state-of-the-art uncertainty calibration across multiple downstream tasks, including cross-modal retrieval, visual question answering, and active learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。