提升视觉语言模型提示调优的置信度校准，让预测更可靠。
Towards Calibrating Prompt Tuning of Vision-Language Models
- 用两个正则项改进损失函数，稳定类别间得分差距。
- 在11个数据集上显著降低预期校准误差（ECE）。
- 适合需要可信置信度的下游任务应用。
大规模视觉语言模型（如CLIP）的提示调优可实现无需更新权重的任务适配，但常导致置信度校准不佳和预测不确定性不可靠。本文提出一种校准框架，在保持预训练CLIP嵌入空间几何结构的前提下提升预测可靠性。方法在标准交叉熵损失基础上引入两项互补正则项：(1) 均值-方差边界惩罚，通过最大化平均对数得分差并最小化其方差来稳定类间得分间距，缓解欠自信与过自信问题；(2) 文本均值匹配损失，使调优后文本嵌入的一阶与二阶矩与冻结的CLIP对应嵌入一致，保留语义分散性以保障泛化能力。在7种提示调优方法和11个多样化数据集上的大量实验表明，该方法相比现有校准技术显著降低了预期校准误差（ECE），涵盖基础类与新类别。
原文摘要 · Abstract (English)
Prompt tuning of large-scale vision-language models such as CLIP enables efficient task adaptation without updating model weights. However, it often leads to poor confidence calibration and unreliable predictive uncertainty. We address this problem by proposing a calibration framework that enhances predictive reliability while preserving the geometry of the pretrained CLIP embedding space, which is required for robust generalization. Our approach extends the standard cross-entropy loss with two complementary regularizers: (1) a mean-variance margin penalty that stabilizes inter-class logit margins by maximizing their average while minimizing dispersion, mitigating underconfidence and overconfidence spikes; and (2) a text moment-matching loss that aligns the first and second moments of tuned text embeddings with their frozen CLIP counterparts, preserving semantic dispersion crucial for generalization. Through extensive experiments across 7 prompt-tuning methods and 11 diverse datasets, we demonstrate that our approach significantly reduces the Expected Calibration Error (ECE) compared to competitive calibration techniques on both base and novel classes
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。