arXiv:2503.12866cs.CV2025-03CVPR被引 11

通过视觉相似性构建样本簇,生成细粒度文本提示,提升视觉语言模型在分布偏移下的测试时适应能力。

SCAP: Transductive Test-Time Adaptation via Supportive Clique-based Attribute Prompting

  • 基于视觉相似性自动构建测试样本簇,统一学习共享属性提示
  • 在多个基准上显著超越现有方法,跨域泛化性能提升明显
  • 适合需要批量测试时动态适应的视觉语言模型部署场景

视觉语言模型(VLMs)在面对数据分布变化引起的领域偏移时面临巨大挑战。测试时适应(TTA)已成为提升VLM在该条件下性能的有前景方法。实际中,测试数据常以批形式到达,促使对归纳式TTA的关注。然而,现有方法多聚焦单个样本,忽视批次内样本间的关联。尽管近期基于ViT的TTA方法引入了批次级适应,但因未能有效融合文本模态,对VLM仍不理想。为此,我们提出一种新型归纳式TTA框架——支持性团块属性提示(SCAP),通过生成细粒度属性提示,有效结合视觉与文本信息以增强适应。SCAP首先基于视觉相似性无监督地形成测试样本的支持性团块,并为每个团块学习属性提示,捕捉关键适应特征。对于每个测试样本,聚合其关联团块的属性提示,提供丰富上下文信息。为确保随时间可适应,引入保留模块,动态更新属性提示及其关联属性。在多个基准上的全面实验表明,SCAP显著优于现有最先进方法,大幅推进了VLM在领域偏移下的泛化能力。代码已公开于https://github.com/zhoujiahuan1991/CVPR2025-SCAP。

原文摘要 · Abstract (English)

Vision-language models (VLMs) encounter considerable challenges when adapting to domain shifts stemming from changes in data distribution. Test-time adaptation (TTA) has emerged as a promising approach to enhance VLM performance under such conditions. In practice, test data often arrives in batches, leading to increasing interest in the transductive TTA setting. However, existing TTA methods primarily focus on individual test samples, overlooking crucial cross-sample correlations within a batch. While recent ViT-based TTA methods have introduced batch-level adaptation, they remain suboptimal for VLMs due to inadequate integration of the text modality. To address these limitations, we propose a novel transductive TTA framework, Supportive Clique-based Attribute Prompting (SCAP), which effectively combines visual and textual information to enhance adaptation by generating fine-grained attribute prompts across test batches. SCAP first forms supportive cliques of test samples in an unsupervised manner based on visual similarity and learns an attribute prompt for each clique, capturing shared attributes critical for adaptation. For each test sample, SCAP aggregates attribute prompts from its associated cliques, providing enriched contextual information. To ensure adaptability over time, we incorporate a retention module that dynamically updates attribute prompts and their associated attributes as new data arrives. Comprehensive experiments across multiple benchmarks demonstrate that SCAP outperforms existing state-of-the-art methods, significantly advancing VLM generalization under domain shifts. Our code is available at https://github.com/zhoujiahuan1991/CVPR2025-SCAP.

测试时适应视觉语言模型属性提示批次适应

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。