用视觉语言模型提升图像分类的不确定性估计，让置信度更精准。
Localized Conformal Prediction for Image Classification with Vision-Language Models
- 基于测试样本与校准样本的相似性，动态调整预测集
- 新方法使平均预测集大小显著缩小，且保持覆盖概率保证
- 适合需要可靠不确定性的图像分类场景
置信区间预测在不确定性量化领域备受关注，主要因其强边际覆盖保证。完全条件覆盖难以实现，这是置信区间预测领域的公认事实。因此，已有方法尝试通过根据测试样本与校准样本的相似性来调整其置信集，以近似条件覆盖行为。尽管该方法在回归任务中表现优异，但在图像分类中的应用仍不充分。本文在自然图像分类任务上，使用开源实现的最新局部化置信区间预测算法，对视觉语言模型（VLMs）进行了广泛基准测试。结果表明，直接使用测试时与校准样本视觉特征间的余弦相似度——一种直观的选择——不足以优于非局部基线。为此，我们提出一种简单的非线性变换对余弦相似度进行处理，既保留了边际覆盖保证，又实现了统计显著的平均预测集大小减少。代码已公开于 https://github.com/cfuchs2023/lcp-vlm/。
原文摘要 · Abstract (English)
Conformal predictions have attracted significant attention in the field of uncertainty quantification, mainly because of their strong marginal coverage guarantees. Full conditional guarantee is not an attainable goal, a well known fact in conformal predictions literature. As a result, several approaches have tried to approximate this behavior by adapting the conformal sets of test-time samples according to their similarity to calibration examples. Although the latter has gained traction and shown impressive performances for regression problems, its application to image classification remains under-explored. We conduct an extensive benchmarking on natural image classification tasks with vision-language models (VLMs), using our open source implementation of a recent localized conformal prediction algorithm. We show that straightforward usage of the cosine similarity between test-time and calibration visual features, an intuitive choice for VLMs, is not sufficient to improve over the non-local baselines. In response, we propose a simple non-linear transformation of the cosine similarities, which conserves marginal coverage guarantees and achieves statistically significant mean set sizes reduction. Code is available at https://github.com/cfuchs2023/lcp-vlm/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。