解决视觉语言模型微调后分类置信度不准问题
Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Models
- 通过动态异常值正则化，平衡基础与新类别的置信度
- 在基类上缓解过低置信度，新类上避免过度自信
- 适用于需高置信度可靠性的视觉语言应用
置信度校准对机器学习模型在现实世界中的安全部署至关重要。然而，像CLIP这样的视觉语言模型在微调后存在的校准问题尚未得到充分解决。本文发现，现有提示调优方法在基类与新类别间存在校准权衡：CoOp的交叉熵损失使新类别产生过度自信，因文本标签差异增大；而KgCoOp的正则化虽维持置信度水平，却因准确率提升导致基类出现置信度不足。受此启发，我们提出动态异常值正则化（DOR），确保微调后基类与新类均保持良好校准。具体而言，DOR最小化从大规模词表中采样的新文本标签的特征偏差，从而防止新标签文本差异增大，同时放松对基类的约束。大量实验表明，DOR能有效提升当前微调方法在基类和新类上的校准性能。
原文摘要 · Abstract (English)
Confidence calibration is critical for the safe deployment of machine learning models in the real world. However, such issue in vision-language models like CLIP, particularly after fine-tuning, has not been fully addressed. In this work, we demonstrate that existing prompt tuning methods usually lead to a trade-off of calibration between base and new classes: the cross-entropy loss in CoOp causes overconfidence in new classes by increasing textual label divergence, whereas the regularization of KgCoOp maintains the confidence level but results in underconfidence in base classes due to the improved accuracy. Inspired by the observations, we introduce Dynamic Outlier Regularization (DOR) to ensure the confidence calibration on both base and new classes after fine-tuning. In particular, we propose to minimize the feature deviation of novel textual labels (instead of base classes) sampled from a large vocabulary. In effect, DOR prevents the increase in textual divergence for new labels while easing restrictions on base classes. Extensive experiments demonstrate that DOR can enhance the calibration performance of current fine-tuning methods on base and new classes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。