用图像文本对齐差异校准模型置信度，提升细调后CLIP在新类别上的可靠性。
Shift-Aware Calibration for Fine-Tuned CLIP: Leveraging Image-Text Alignment
- 通过原模型与微调后输出逻辑斯蒂差异作为校准信号
- 在11个数据集上改善训练/未见类的置信度准确性
- 无需训练，适合跨数据集和领域泛化的场景
视觉-语言模型（如CLIP）通过提示调优可有效适应下游任务，但微调可能导致预测置信度与准确率失配，尤其在未见类别上。现有针对VLM的校准方法主要依赖训练类别的文本特征，限制了其在类别和分布漂移下的适用性。我们提出无需训练的样本级校准方法—— extbf{Shift-Aware Calibration (SAC)}，利用原始模型与微调后模型输出逻辑斯蒂之间的差异作为校准信号。该量称为“逻辑斯蒂偏移”（logit shift），因其在预测空间中度量而非隐藏表示间。SAC将此偏移映射为正缩放因子，在不改变预测类别前提下调整置信度。在11个数据集及五种微调方法下，SAC均提升了训练类和未见类的校准效果，并在跨数据集与领域泛化评估中表现优异。互信息（MI）分析表明，逻辑斯蒂偏移信号与期望校准误差（ECE）存在统计相关性，为该信号的信息有效性提供了实证支持，验证了其设计的合理性。
原文摘要 · Abstract (English)
Vision-language models (VLMs), such as CLIP, adapt effectively to downstream tasks through prompt tuning, but fine-tuning can misalign predictive confidence and accuracy, particularly on unseen classes. Existing VLM-specific calibration methods mainly rely on textual features of train classes, limiting their applicability across class and distribution shifts. We propose \textbf{Shift-Aware Calibration (SAC)}, a training-free, sample-wise method that uses the discrepancy between the output logits of the original and fine-tuned CLIP as a calibration signal. We explicitly call this quantity \emph{logit shift}, since it is measured in prediction space rather than between hidden representations. SAC maps this logit shift to a positive scaling factor that adjusts confidence while preserving the predicted class. Across 11 datasets and five fine-tuning methods, SAC improves calibration on train and unseen classes and under cross-dataset and domain-generalization evaluation. Mutual Information (MI) analysis shows a statistical dependency between the logit-shift signal and ECE across methods and datasets, providing empirical evidence that the signal used by SAC is informative for calibration and supporting the effectiveness of its shift-aware design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。