无需微调，用类内特征一致性检测视觉语言模型的错误预测。
Intra-Class Probabilistic Embeddings for Uncertainty Estimation in Vision-Language Models
- 通过多变量高斯建模类内特征分布，生成类专属概率嵌入。
- 仅需每类10张图，在多个数据集上误差检测性能领先。
- 适用于安全关键场景，对分布偏移鲁棒，可适配任意对比式VLM。
视觉语言模型（如CLIP）虽具备强大开放词汇分类能力，但易对错误分类赋予高置信度，限制其在安全关键应用中的可靠性。本文提出一种无需训练、后处理的不确定性估计方法，用于检测对比式视觉语言模型的错误预测。核心思想是通过特征投影与多变量高斯分布，度量类别内部视觉特征的一致性，构建类专属的概率嵌入。该方法不依赖特定模型，无需微调，对分布偏移具有鲁棒性，且在每类仅10张训练图像时仍表现良好。在ImageNet、Flowers102、Food101、EuroSAT和DTD上的实验表明，其误差检测性能显著优于各类确定性和概率性基线。代码已公开于https://github.com/zhenxianglin/ICPE。
原文摘要 · Abstract (English)
Vision-language models (VLMs), such as CLIP, have gained popularity for their strong open vocabulary classification performance, but they are prone to assigning high confidence scores to misclassifications, limiting their reliability in safety-critical applications. We introduce a training-free, post-hoc uncertainty estimation method for contrastive VLMs that can be used to detect erroneous predictions. The key to our approach is to measure visual feature consistency within a class, using feature projection combined with multivariate Gaussians to create class-specific probabilistic embeddings. Our method is VLM-agnostic, requires no fine-tuning, demonstrates robustness to distribution shift, and works effectively with as few as 10 training images per class. Extensive experiments on ImageNet, Flowers102, Food101, EuroSAT and DTD show state-of-the-art error detection performance, significantly outperforming both deterministic and probabilistic VLM baselines. Code is available at https://github.com/zhenxianglin/ICPE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。