让CLIP模型在测试时自动适应图像损坏，提升识别鲁棒性。
$\texttt{BATCLIP}$: Bimodal Online Test-Time Adaptation for CLIP
- 利用双模态在线适配，同时优化图像和文本特征对齐
- 在ImageNet-C上显著提升CLIP在图像损坏下的准确率
- 适合需要高鲁棒性的实际部署场景
尽管对比语言图像预训练（CLIP）等开放词汇分类模型展现出强大的零样本学习能力，但其在测试时面对常见图像损坏的鲁棒性仍不明确。通过大量实验，我们发现零样本CLIP在测试时对常见图像损坏缺乏鲁棒性，亟需使用测试时自适应（TTA）方法对未标注的损坏图像进行适配。然而，现有TTA方法因单模态特性，难以有效适配CLIP。为此，我们提出一种双模态在线TTA方法BATCLIP，旨在提升CLIP对常见图像损坏的鲁棒性。核心思想是不仅优化视觉编码器以改善图像特征，还通过伪标签计算的图像类别原型，增强图像与文本特征之间的关联。我们在基准图像损坏数据集上评估该方法，在CLIP的在线TTA中取得领先性能。此外，我们在多个域泛化数据集上验证了该方法的泛化能力。代码已开源。
原文摘要 · Abstract (English)
Although open-vocabulary classification models like Contrastive Language Image Pretraining (CLIP) have demonstrated strong zero-shot learning capabilities, their robustness to common image corruptions remains poorly understood. Through extensive experiments, we show that zero-shot CLIP lacks robustness to common image corruptions during test-time, necessitating the adaptation of CLIP to unlabeled corrupted images using test-time adaptation (TTA). However, we found that existing TTA methods have severe limitations in adapting CLIP due to their unimodal nature. To address these limitations, we propose $\texttt{BATCLIP}$, a bimodal $\textbf{online}$ TTA method designed to improve CLIP's robustness to common image corruptions. The key insight of our approach is not only to adapt the visual encoders for improving image features but also to strengthen the alignment between image and text features by promoting a stronger association between the image class prototype, computed using pseudo-labels, and the corresponding text feature. We evaluate our approach on benchmark image corruption datasets and achieve state-of-the-art results in online TTA for CLIP. Furthermore, we evaluate our proposed TTA approach on various domain generalization datasets to demonstrate its generalization capabilities. Our code is available at https://github.com/sarthaxxxxx/BATCLIP
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。