arXiv:2608.29395cs.CV2026-08中稿 · the British Machin…

用可靠样本生成双高斯证据,提升冻结视觉语言模型的零样本鲁棒性。

GATE: Reliability-Gated Gaussian Evidence Fusion for Training-Free Test-Time Adaptation of Vision-Language Models

  • 构建文本与图像双高斯证据,通过可靠性门控控制图像伪证据影响。
  • 在多个数据集上平均提升5.41分,超越最强基线1.94分。
  • 无需训练、不修改模型参数,适合部署时快速适配新场景。

视觉语言模型如CLIP和SigLIP具备强大的零样本识别能力,但在目标数据分布偏离预训练分布时性能会下降。测试时自适应提供了一种无需源数据或目标标签即可提升鲁棒性的实用方法,但现有方法通常仅依赖提示侧或图像侧的单一证据。本文提出GATE,一种训练-free的两阶段归纳式测试时自适应框架,利用未标注目标集,同时保持图像编码器、文本编码器和提示参数完全冻结。GATE在共享视觉-语言特征空间中构建两个互补的高斯证据源:基于多语言描述的文本高斯和基于可靠未标注目标样本的图像高斯。类别级可靠性门控调节图像伪证据的影响,得分级广义专家乘积融合生成归一化残差修正原始零样本逻辑值。在细粒度识别数据集、ImageNet族分布偏移、多个CLIP骨干网络及SigLIP-B/16上,GATE在每一组基准/骨干结构中均取得最佳平均准确率。其平均提升零样本性能5.41点,优于最强非GATE基线1.94点,证明了可靠性门控分布证据在冻结视觉语言模型自适应中的有效性。

原文摘要 · Abstract (English)

Vision-language models such as CLIP and SigLIP provide strong zero-shot recognition, but their predictions can degrade when deployed on target data that differ from the pretraining distribution. Test-time adaptation offers a practical way to improve robustness without source data or target labels, yet existing methods often rely on either prompt-side adaptation or image-side target evidence alone. In this work, we introduce GATE, a training-free two-pass transductive test-time adaptation framework that uses the unlabeled target set while keeping the image encoder, text encoder, and prompt parameters fully frozen. Instead of representing each class with a single prototype, GATE builds two complementary Gaussian sources of evidence in the shared vision-language feature space: a text Gaussian estimated from multiple language descriptions and an image Gaussian estimated from reliable unlabeled target samples. A class-wise reliability gate controls the influence of image-derived pseudo-evidence, and a score-level generalized Product-of-Experts fusion produces a normalized residual correction to the original zero-shot logits. Across fine-grained recognition datasets, ImageNet-family distribution shifts, multiple CLIP backbones, and SigLIP-B/16, GATE achieves the best average accuracy in every benchmark/backbone group. It improves zero-shot performance by an average of 5.41 points and outperforms the strongest non-GATE baseline by 1.94 points, demonstrating the benefit of reliability-gated distributional evidence for frozen VLM adaptation.

视觉语言模型测试时自适应零样本学习高斯融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。