零样本视觉语言模型在分布偏移下,边缘覆盖率不能保证类别尾部安全。
Does Marginal Coverage Guarantee Class-Conditional Safety for Zero-Shot VLMs Under Shift?
- 用分拆校准预测法提升边缘覆盖,但忽略类别条件下的尾部风险。
- 在图像草图数据集上,最差类别的覆盖率降至约0,10-12%类低于置信下限。
- 仅目标域标签可恢复最差类别保护,但代价高且不适用于无标签场景。
分拆校准预测在交换性假设下可提供边缘覆盖,现被广泛用于零样本视觉语言模型(如CLIP、OpenCLIP、SigLIP)的拒答层。本文在ImageNet及非ImageNet设置下评估其在部署分布偏移下的表现。结果显示:尽管边缘覆盖率保持在约0.86,但类别条件下的尾部覆盖率严重崩溃——在ImageNet-Sketch上,最差类覆盖率降至≈0,10-12%的类别低于有限样本零假设下界。该失败与目标域分类准确率相关,但无法通过源域诊断预测。源域Mondrian校准仅改善分布内尾部,不可迁移;聚类校准和Conf-OT提升平均或边缘指标,但无法恢复最差类尾部。目标域类别校准显著提升尾部性能,但需每类标注且计算量大。此外发现跨模型家族存在2-3倍效率差距,且原生SigLIP sigmoid得分破坏了APS的概率质量解释。上述现象在不同模型规模、预训练语料、提示方式、误覆盖水平α及非ImageNet偏移设置下均持续存在。因此,边缘覆盖应视为平均可靠性指标,而非类别尾部的安全保障。
原文摘要 · Abstract (English)
Split-conformal prediction provides marginal coverage under exchangeability and is increasingly used as an abstention layer for zero-shot vision-language models (VLMs). We audit this practice under deployment shift for CLIP, OpenCLIP, and SigLIP across ImageNet and non-ImageNet settings. Marginal coverage can remain relatively high while class-conditional tail coverage collapses: on ImageNet-Sketch, worst-class coverage falls to $\approx 0$ and 10-12% of classes lie below a finite-sample null floor, despite marginal coverage of about 0.86. The failure is aligned with target-domain class accuracy but is not predicted by the source-domain diagnostics we test. Source-side Mondrian calibration improves the in-distribution tail but does not transfer, while clustered conformal and Conf-OT improve marginal or average metrics without recovering the worst-class tail. Target-side class calibration substantially lifts the tail, but requires labels for every class and remains set-size-intensive. We further identify a 2-3$\times$ cross-family efficiency gap and show that native SigLIP sigmoid scores remove APS's probability-mass interpretation. The findings persist across the tested model scale, pretraining corpus, prompt, miscoverage level $α$, and shifted non-ImageNet settings. Marginal conformal coverage should therefore be treated as an average reliability statistic, not as a safety guarantee for the class tail.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。