提出更真实的视觉语言模型测试时自适应方法，提升实际部署鲁棒性。
Realistic Test-Time Adaptation of Vision-Language Models
- 设计新评估框架，支持批量中有效类别数可变、测试样本非独立同分布。
- 发现现有方法在理想条件下表现好，但会损害模型初始零样本泛化能力。
- 提出StatA方法，通过统计锚定正则项保护文本编码器知识，适配复杂场景。
视觉语言模型（VLMs）的零样本能力被广泛用于提升预测性能。然而，以往的归纳或测试时自适应（TTA）方法常假设数据分布理想，如所有类别均存在。本文挑战这些理想化部署场景，提出更真实的评估框架：(i) 单批次内有效类别数可变；(ii) 在在线自适应设置下测试样本非独立同分布。全面评估与消融实验表明，当前的归纳或TTA方法在有利假设下虽能提升性能，却系统性削弱模型在多种真实场景下的初始零样本鲁棒性。为此，我们提出StatA，一种能应对多种部署场景的通用方法，特别在低数据条件下，引入专为VLM设计的新正则项，作为统计锚点以保留初始文本编码器知识。代码已公开于 https://github.com/MaxZanella/StatA。
原文摘要 · Abstract (English)
The zero-shot capabilities of Vision-Language Models (VLMs) have been widely leveraged to improve predictive performance. However, previous works on transductive or test-time adaptation (TTA) often make strong assumptions about the data distribution, such as the presence of all classes. Our work challenges these favorable deployment scenarios, and introduces a more realistic evaluation framework, including: (i) a variable number of effective classes for adaptation within a single batch, and (ii) non-i.i.d. batches of test samples in online adaptation settings. We provide comprehensive evaluations, comparisons, and ablation studies that demonstrate how current transductive or TTA methods for VLMs systematically compromise the models' initial zero-shot robustness across various realistic scenarios, favoring performance gains under advantageous assumptions about the test samples' distributions. Furthermore, we introduce StatA, a versatile method that could handle a wide range of deployment scenarios, including those with a variable number of effective classes at test time. Our approach incorporates a novel regularization term designed specifically for VLMs, which acts as a statistical anchor preserving the initial text-encoder knowledge, particularly in low-data regimes. Code available at https://github.com/MaxZanella/StatA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。