提出零样本熵校准法,让测试时适配的视觉语言模型预测更可信。
Respect Your Zero-Shot Uncertainty: Conservative Calibration for Test-Time-Adapted Vision-Language Models
- 用零样本熵做不确定性参考,仅通过温度缩放修复过度自信
- 在15个数据集上降低30%以上预期校准误差,保持准确率不变
- 无需标签数据或训练参数,适合部署阶段提升模型可靠性
测试时适配(TTA)虽能提升视觉语言模型在分布外场景下的识别准确率,但常导致校准性下降,使预测置信度不可靠。现有无标签校准方法多依赖提示优化或对数范围统计,表征粗略。我们发现,即使最优预测和正确性不变,TTA仍可能提升置信度、降低熵,此为‘预测保持锐化’。熵减少越多,预期校准误差(ECE)上升越明显;在熵降低的样本上,置信度增益常超过准确率增益。为此提出零样本锚定熵校准(ZAEC),以零样本熵作为样本级不确定性基准,仅对锐化预测进行最小温度缩放恢复其熵,其余保持不变。该方法无需标签校准数据或学习参数,保留类别排序与分类准确率。在五种TTA方法、15个数据集上,对ViT-B/16实现最低后处理宏观平均ECE,RN50也持续增益。
原文摘要 · Abstract (English)
Test-time adaptation (TTA) can improve the recognition accuracy of vision-language models under distribution shift, but often degrades calibration, making predictive confidence unreliable for downstream decision-making. Many existing label-free calibration approaches are either coupled to prompt optimization or rely on logit-range statistics that provide only a coarse characterization of the predictive distribution. We show that TTA can increase confidence and reduce entropy even when the top-1 prediction and its correctness remain unchanged, a failure mode we term prediction-preserving sharpening. Across diverse TTA methods and benchmarks, larger entropy reductions relative to paired zero-shot predictions are associated with greater increases in Expected Calibration Error (ECE). On entropy-reduced samples, confidence gains also tend to exceed accuracy gains. Based on these findings, we propose Zero-Shot-Anchored Entropy Calibration (ZAEC), a label-free post-hoc method that uses zero-shot entropy as a sample-specific uncertainty reference. ZAEC selectively restores the zero-shot entropy of sharpened predictions through minimal temperature scaling while leaving all other predictions unchanged. It requires no labeled calibration data or learned parameters and preserves class rankings and classification accuracy. Across five TTA methods and 15 datasets, ZAEC achieves the lowest post-hoc macro-average ECE on ViT-B/16, with consistent gains on RN50.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。