arXiv:2409.19375cs.LGcs.AI2024-09NeurIPS被引 23

让视觉语言模型在测试时自动适应数据分布变化,避免记忆丢失。

DOTA: Distributional Test-Time Adaptation of Vision-Language Models

  • 用贝叶斯方法动态估计测试数据分布,而非简单存储样本。
  • 在多个数据集上显著减少遗忘,性能超越现有方法。
  • 适合部署在分布漂移场景下的模型,如真实世界应用。

视觉语言基础模型(如CLIP)在多种任务中表现优异,但在训练与测试数据存在显著分布差异时部署可靠性下降,而针对多样场景微调成本高昂。基于缓存的测试时适配器通过存储代表性测试样本实现高效适配,但通常采用简单缓存管理,容量有限,导致更新时样本被丢弃引发严重灾难性遗忘。本文提出DOTA(Distributional Test-time Adaptation),一种简单却有效的方法。关键在于不单纯记忆单个测试样本,而是持续估计测试数据流的底层分布,利用贝叶斯定理计算测试时后验概率进行模型适配。该以分布为中心的方法使模型能持续学习并适应部署环境。大量实验表明,DOTA显著缓解遗忘问题,在多个基准上达到当前最优性能。

原文摘要 · Abstract (English)

Vision-language foundation models (VLMs), such as CLIP, exhibit remarkable performance across a wide range of tasks. However, deploying these models can be unreliable when significant distribution gaps exist between training and test data, while fine-tuning for diverse scenarios is often costly. Cache-based test-time adapters offer an efficient alternative by storing representative test samples to guide subsequent classifications. Yet, these methods typically employ naive cache management with limited capacity, leading to severe catastrophic forgetting when samples are inevitably dropped during updates. In this paper, we propose DOTA (DistributiOnal Test-time Adaptation), a simple yet effective method addressing this limitation. Crucially, instead of merely memorizing individual test samples, DOTA continuously estimates the underlying distribution of the test data stream. Test-time posterior probabilities are then computed using these dynamically estimated distributions via Bayes' theorem for adaptation. This distribution-centric approach enables the model to continually learn and adapt to the deployment environment. Extensive experiments validate that DOTA significantly mitigates forgetting and achieves state-of-the-art performance compared to existing methods.

视觉语言模型测试时适配分布外泛化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。