让视觉语言模型在测试时持续学习，提升跨场景泛化能力
Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models
- 双原型演化机制，动态更新文本与视觉原型
- 15个数据集上均超越现有最优方法，精度显著提升
- 适合需要实时适应新场景的视觉语言应用
测试时自适应能够使模型利用无标签测试样本泛化到多样数据，在真实场景中具有重要意义。近期研究将此设定应用于先进的预训练视觉语言模型（VLMs），提出测试时提示调优等方法以拓展其实用性。然而，这些方法通常仅针对单一模态进行适应，无法随着样本积累持续获取任务特定知识。为此，本文提出双原型演化（DPE）方法，通过构建并演化文本与视觉两类原型，逐步捕捉目标类别更准确的多模态表示。为促进模态间一致性，引入可学习残差对每个测试样本进行优化，以对齐双模态原型。在15个基准数据集上的实验表明，DPE在多数场景下持续优于现有最先进方法，同时具备良好的计算效率。代码已开源：https://github.com/zhangce01/DPE-CLIP。
原文摘要 · Abstract (English)
Test-time adaptation, which enables models to generalize to diverse data with unlabeled test samples, holds significant value in real-world scenarios. Recently, researchers have applied this setting to advanced pre-trained vision-language models (VLMs), developing approaches such as test-time prompt tuning to further extend their practical applicability. However, these methods typically focus solely on adapting VLMs from a single modality and fail to accumulate task-specific knowledge as more samples are processed. To address this, we introduce Dual Prototype Evolving (DPE), a novel test-time adaptation approach for VLMs that effectively accumulates task-specific knowledge from multi-modalities. Specifically, we create and evolve two sets of prototypes--textual and visual--to progressively capture more accurate multi-modal representations for target classes during test time. Moreover, to promote consistent multi-modal representations, we introduce and optimize learnable residuals for each test sample to align the prototypes from both modalities. Extensive experimental results on 15 benchmark datasets demonstrate that our proposed DPE consistently outperforms previous state-of-the-art methods while also exhibiting competitive computational efficiency. Code is available at https://github.com/zhangce01/DPE-CLIP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。