针对视觉语言模型在分布偏移下的泛化难题，提出一种对齐预训练目标的测试时自适应方法。
CLIPTTA: Robust Contrastive Vision-Language Test-Time Adaptation
- 基于软对比损失，使测试时更新与CLIP的预训练目标一致
- 在75个数据集上性能超越熵最小化基线，且更稳定
- 适用于包含异常样本的开放集场景，提升异常检测能力
视觉语言模型（如CLIP）具备强大的零样本能力，但在分布偏移下常表现不佳。测试时自适应（TTA）可在推理阶段无标签数据条件下进行模型更新，通常采用熵最小化目标。然而该目标与视觉语言模型的对比学习预训练目标本质不一致，限制了适应性能，并引发伪标签漂移和类别坍塌等失败模式。本文提出CLIPTTA，一种基于梯度的视觉语言模型TTA方法，利用与CLIP预训练目标对齐的软对比损失。我们提供了CLIPTTA梯度的理论分析，表明其批次感知设计能有效缓解坍塌风险。进一步将CLIPTTA扩展至开放集场景，引入异常对比暴露（OCE）损失以增强对分布外（OOD）样本的检测能力。在涵盖多样分布偏移的75个数据集上评估显示，CLIPTTA持续优于熵基方法，且在多数数据集上超越当前最优TTA方法，表现出更稳健的性能。
原文摘要 · Abstract (English)
Vision-language models (VLMs) like CLIP exhibit strong zero-shot capabilities but often fail to generalize under distribution shifts. Test-time adaptation (TTA) allows models to update at inference time without labeled data, typically via entropy minimization. However, this objective is fundamentally misaligned with the contrastive image-text training of VLMs, limiting adaptation performance and introducing failure modes such as pseudo-label drift and class collapse. We propose CLIPTTA, a new gradient-based TTA method for vision-language models that leverages a soft contrastive loss aligned with CLIP's pre-training objective. We provide a theoretical analysis of CLIPTTA's gradients, showing how its batch-aware design mitigates the risk of collapse. We further extend CLIPTTA to the open-set setting, where both in-distribution (ID) and out-of-distribution (OOD) samples are encountered, using an Outlier Contrastive Exposure (OCE) loss to improve OOD detection. Evaluated on 75 datasets spanning diverse distribution shifts, CLIPTTA consistently outperforms entropy-based objectives and is highly competitive with state-of-the-art TTA methods, outperforming them on a large number of datasets and exhibiting more stable performance across diverse shifts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。