测试时动态调整文本特征，提升视觉语言模型泛化能力
Test-time Alignment-Enhanced Adapter for Vision-Language Models
- 引入TAEA适配器，在测试时动态优化文本特征
- 在分布外和跨域任务上分别提升0.75%和2.5%
- 适合需要强泛化能力的视觉语言应用
基于预训练视觉语言模型（VLMs）的测试时自适应（TTA）方法近年来受到关注，用于缓解测试阶段的分布偏移问题。现有方法虽能通过调整分类逻辑值缓解该问题，但因保持文本特征不变而效果受限。为此，本文提出测试时对齐增强适配器（TAEA），通过使用测试样本训练适配器，在测试阶段动态调整文本特征，从而增强文本与图像的对齐预测。此外，还引入来自TDA的负样本缓存作为增强模块，进一步提升性能。实验表明，该方法在分布外基准上平均超越当前最优TTA方法0.75%，在跨域基准上提升2.5%，且训练开销可接受。代码将开源于https://github.com/BaoshunWq/clip-TAEA。
原文摘要 · Abstract (English)
Test-time adaptation with pre-trained vision-language models (VLMs) has attracted increasing attention for tackling the issue of distribution shift during the test phase. While prior methods have shown effectiveness in addressing distribution shift by adjusting classification logits, they are not optimal due to keeping text features unchanged. To address this issue, we introduce a new approach called Test-time Alignment-Enhanced Adapter (TAEA), which trains an adapter with test samples to adjust text features during the test phase. We can enhance the text-to-image alignment prediction by utilizing an adapter to adapt text features. Furthermore, we also propose to adopt the negative cache from TDA as enhancement module, which further improves the performance of TAEA. Our approach outperforms the state-of-the-art TTA method of pre-trained VLMs by an average of 0.75% on the out-of-distribution benchmark and 2.5% on the cross-domain benchmark, with an acceptable training time. Code will be available at https://github.com/BaoshunWq/clip-TAEA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。