arXiv:2510.26441cs.CV2025-10中稿 · ICLR被引 3

通过提升文本特征的角向多样性,增强视觉语言模型测试时提示调优的校准能力。

A-TPT: Angular Diversity Calibration Properties for Test-Time Prompt Tuning of Vision-Language Models

  • 引入角向多样性最大化机制,使文本特征在单位超球面上均匀分布。
  • 在多个数据集上降低综合平均校准误差,且保持相近准确率。
  • 尤其擅长处理自然分布偏移和医学数据,适合高可靠性场景使用。

测试时提示调优(TPT)已成为无需标注数据即可适应大型视觉语言模型(VLMs)新任务的有前景技术。然而,文本特征缺乏分散性会损害校准性能,影响VLMs的可靠性、可信度与安全性。现有TPT方法主要通过最大化平均文本特征分散性或施加正交约束来改善校准,但未必能实现类别间特征的最佳角分离,忽略了角向多样性的关键作用。为此,我们提出A-TPT,一种新颖的TPT框架,通过最大化单位超球面上特征间的最小成对角距离,促进可学习提示所诱导的归一化文本特征分布的均匀性。实验表明,该方法在多种骨干网络和数据集上均显著降低聚合平均校准误差,同时保持相当的准确性。特别地,在自然分布偏移下的零样本校准表现优异,并能良好泛化至医疗数据集。我们进行了广泛的分析,包括理论论证,证实了A-TPT的合理性。结果表明,提升角向多样性有助于获得更分散的文本特征,显著改善测试时适配中的校准效果。代码将公开发布。

原文摘要 · Abstract (English)

Test-time prompt tuning (TPT) has emerged as a promising technique for adapting large vision-language models (VLMs) to unseen tasks without relying on labeled data. However, the lack of dispersion between textual features can hurt calibration performance, which raises concerns about VLMs' reliability, trustworthiness, and safety. Current TPT approaches primarily focus on improving prompt calibration by either maximizing average textual feature dispersion or enforcing orthogonality constraints to encourage angular separation. However, these methods may not always have optimal angular separation between class-wise textual features, which implies overlooking the critical role of angular diversity. To address this, we propose A-TPT, a novel TPT framework that introduces angular diversity to encourage uniformity in the distribution of normalized textual features induced by corresponding learnable prompts. This uniformity is achieved by maximizing the minimum pairwise angular distance between features on the unit hypersphere. We show that our approach consistently surpasses state-of-the-art TPT methods in reducing the aggregate average calibration error while maintaining comparable accuracy through extensive experiments with various backbones on different datasets. Notably, our approach exhibits superior zero-shot calibration performance on natural distribution shifts and generalizes well to medical datasets. We provide extensive analyses, including theoretical aspects, to establish the grounding of A-TPT. These results highlight the potency of promoting angular diversity to achieve well-dispersed textual features, significantly improving VLM calibration during test-time adaptation. Our code will be made publicly available.

视觉语言模型提示调优校准角向多样性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。