arXiv:2507.11969cs.CV2025-07ICML被引 1

通过两种可学习偏置提升视觉语言模型测试时适应性能,高效且效果领先。

GS-Bias: Global-Spatial Bias Learner for Single-Image Test-Time Adaptation of Vision-Language Models

  • 引入全局与空间偏置,从图像多视角中学习语义一致性与区域关联性。
  • 在15个数据集上达最优表现,跨数据集泛化提升2.23%,域泛化提升2.72%。
  • 无需反向传播,仅需6.5%内存开销,适合资源受限场景部署。

视觉语言模型(VLMs)的测试时自适应(TTA)近期受到广泛关注,尤其依赖单图多增强视图以提升零样本泛化能力。然而,现有方法难以兼顾性能与效率:或因文本提示调优带来过重开销,或因手工设计、无需训练的视觉特征增强效果不稳定。本文提出全局-空间偏置学习器(GS-Bias),一种高效且有效的TTA范式,在测试时引入两个可学习偏置——全局偏置与空间偏置。全局偏置通过学习多视图间的一致性,捕捉测试图像的全局语义特征;空间偏置则学习图像空间表示中各区域间的语义连贯性。这两个偏置直接叠加于预训练VLM的输出logits上,避免了对整个VLM进行反向传播,显著提升效率。实验表明,该方法在15个基准数据集上均达到最先进性能,例如在跨数据集泛化上优于TPT 2.23%,在域泛化上提升2.72%,同时在ImageNet上仅需TPT 6.5%的内存占用。

原文摘要 · Abstract (English)

Recent advances in test-time adaptation (TTA) for Vision-Language Models (VLMs) have garnered increasing attention, particularly through the use of multiple augmented views of a single image to boost zero-shot generalization. Unfortunately, existing methods fail to strike a satisfactory balance between performance and efficiency, either due to excessive overhead of tuning text prompts or unstable benefits from handcrafted, training-free visual feature enhancement. In this paper, we present Global-Spatial Bias Learner (GS-Bias), an efficient and effective TTA paradigm that incorporates two learnable biases during TTA, unfolded as the global bias and spatial bias. Particularly, the global bias captures the global semantic features of a test image by learning consistency across augmented views, while spatial bias learns the semantic coherence between regions in the image's spatial visual representation. It is worth highlighting that these two sets of biases are directly added to the logits outputed by the pretrained VLMs, which circumvent the full backpropagation through VLM that hinders the efficiency of existing TTA methods. This endows GS-Bias with extremely high efficiency while achieving state-of-the-art performance on 15 benchmark datasets. For example, it achieves a 2.23% improvement over TPT in cross-dataset generalization and a 2.72% improvement in domain generalization, while requiring only 6.5% of TPT's memory usage on ImageNet.

测试时自适应视觉语言模型高效学习偏置学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。