提升视觉语言模型在噪声图像下的表现,仅通过测试时优化嵌入分布。
Mint: A Simple Test-Time Adaptation of Vision-Language Models against Common Corruptions

- 通过伪标签最大化类间嵌入方差来增强表示质量。
- 在多种噪声场景下,对CLIP系列模型性能提升显著,最高达15.2%。
- 方法简单高效,小批量即可运行,适合部署在资源受限环境。
预训练的视觉语言模型如CLIP具备强大的零样本泛化能力,但对输入噪声引起的分布偏移仍敏感。本文研究噪声对CLIP图像嵌入的影响,发现一种称为嵌入方差坍缩的现象:随着噪声强度增加,类内和类间方差均减小。该现象与性能下降密切相关,类间方差与分类准确率高度相关。理论分析表明,视觉编码器会捕捉噪声相关信号,稀释类别判别特征并压缩表示几何结构。进一步发现,即使使用伪标签估计,最大化类间方差也能显著提升嵌入质量。基于此,提出Mint——一种简单的测试时自适应方法,通过均值累加器与梯度累加器在线最大化伪标签驱动的类间方差。Mint在小批量下有效,跨多个噪声基准和CLIP架构均稳定提升性能,最高提升达15.2%。代码已开源。
原文摘要 · Abstract (English)
Pretrained vision-language models such as CLIP achieve strong zero-shot generalization but remain vulnerable to distribution shifts caused by input corruptions. In this work, we investigate how corruptions affect CLIP's image embeddings and uncover a consistent phenomenon we term as embedding variance collapse, where both intra-class and inter-class variances shrink as corruption severity increases. We find that this collapse is closely tied to performance degradation, with inter-class variance strongly correlated with classification accuracy. To explain this phenomenon, we analyze how corruptions alter the structure of the embedding space. Our theoretical results suggest that the visual encoder tends to encode corruption-related signals, which dilute class-discriminative features and compress the representation geometry. We further show that maximizing inter-class variance, even when estimated from pseudo-labels, can provably enhance embedding quality. Based on this insight, we propose Mint, a simple test-time adaptation method that maximizes pseudo-label-based inter-class variance on the fly using a mean accumulator and a gradient accumulator. Mint operates effectively with small batch sizes and consistently improves performance across multiple corruption benchmarks and CLIP architectures. Our code is available at https://github.com/baowenxuan/Mint .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。