为视觉语言模型添加后验不确定性估计，无需重新训练
Post-hoc Probabilistic Vision-Language Models
- 在模型最后一层引入贝叶斯近似，分析余弦相似度的不确定性
- 在主动学习中提升预测不确定性校准度，减少样本需求
- 适合需要可信度评估的安全关键场景
视觉语言模型（如 CLIP 和 SigLIP）在分类、检索和生成任务中表现优异，其通过确定性映射将图像与文本编码到联合嵌入空间，并使用余弦相似度衡量匹配程度。然而，这种确定性映射无法捕捉概念不确定性，尤其在领域偏移时影响下游任务表现。本文提出一种无需额外训练的后验不确定性估计方法，基于模型最后几层的贝叶斯后验近似，对余弦相似度进行解析量化。实验表明，该方法在不确定性量化和主动学习中的支持集选择上表现更优，获得更准确且可解释的不确定性估计，实现样本高效主动学习。结果表明其在大规模模型安全应用中具有潜力。
原文摘要 · Abstract (English)
Vision-language models (VLMs), such as CLIP and SigLIP, have found remarkable success in classification, retrieval, and generative tasks. For this, VLMs deterministically map images and text descriptions to a joint latent space in which their similarity is assessed using the cosine similarity. However, a deterministic mapping of inputs fails to capture uncertainties over concepts arising from domain shifts when used in downstream tasks. In this work, we propose post-hoc uncertainty estimation in VLMs that does not require additional training. Our method leverages a Bayesian posterior approximation over the last layers in VLMs and analytically quantifies uncertainties over cosine similarities. We demonstrate its effectiveness for uncertainty quantification and support set selection in active learning. Compared to baselines, we obtain improved and well-calibrated predictive uncertainties, interpretable uncertainty estimates, and sample-efficient active learning. Our results show promise for safety-critical applications of large-scale models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。