提出动态更新嵌入的测试时自适应方法,提升视觉语言模型在分布偏移下的泛化能力。
ETTA: Efficient Test-Time Adaptation for Vision-Language Models through Dynamic Embedding Updates
- 通过递归更新机制融合所有测试样本,动态扩展决策边界。
- 在两个基准上优于现有方法,准确率更高且计算开销极低。
- 适合需要高效自适应的部署场景,如边缘设备或实时系统。
预训练的视觉语言模型(如 CLIP)在零样本任务中表现优异,但在分布偏移下泛化能力不足。测试时自适应(TTA)通过在新域的无标签测试数据上调整模型来缓解此问题。尽管部分方法依赖提示调优,但无需训练的基于缓存的方法更受青睐。然而,现有缓存式 TTA 模型仅存储高置信度样本,限制了决策边界,忽略了其他测试数据的影响。为此,我们提出高效测试时自适应(ETTA),引入递归更新模块,整合所有输入测试样本,逐步优化决策边界。该策略模拟无限缓存,以极小内存和计算开销动态更新上下文嵌入,提升精度。ETTA 还包含自适应集成模块,通过动态选择每类最优提示降低图像-文本分数对提示的依赖。此外,根据置信度自适应融合两模块得分,发挥互补优势。大量实验表明,ETTA 在两个基准上均超越当前最优的 TTA 模型,在计算复杂度与准确率方面树立新标准。代码已开源:https://github.com/hamidreza-dastmalchi/ETTA。
原文摘要 · Abstract (English)
Pretrained vision-language models (VLMs) like CLIP show strong zero-shot performance but struggle with generalization under distribution shifts. Test-Time Adaptation (TTA) addresses this by adapting VLMs to unlabeled test data in new domains. While some TTA methods rely on prompt-tuning, training-free cache-based approaches are preferred for efficiency. However, current cache-based TTA models store only a limited set of high-confidence samples, restricting the decision boundary to these samples and ignoring the influence of other incoming test data. To address this, we propose Efficient Test-Time Adaptation (ETTA), introducing a Recursive Updating module that integrates all incoming test samples, progressively refining the decision boundary. This strategy mimics an unbounded cache, dynamically updating contextual embeddings for improved accuracy with minimal memory and computational overhead. ETTA also includes an Adaptive Ensemble module to reduce prompt dependency in image-to-text scores by dynamically selecting optimal prompts for each class. Furthermore, ETTA adaptively combines scores from both modules based on confidence levels, leveraging their complementary strengths. Extensive experiments on two benchmarks confirm that ETTA surpasses the state-of-the-art TTA models in computational complexity and accuracy, setting a new standard for effective, efficient test-time adaptation. The code has been released at https://github.com/hamidreza-dastmalchi/ETTA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。