用噪声优化视觉输入，让零样本模型更适应新场景。
Noise is an Efficient Learner for Zero-Shot Vision-Language Models
- 直接在视觉输入加可学习噪声，动态调整特征表示。
- 在自然分布上提升7.38%，跨数据集评估提升0.80%。
- 适合需要快速适应新环境的零样本视觉语言任务。
测试时自适应近年受到关注，用于在无标注数据下微调模型。传统方法主要通过调整可学习提示来适应预训练视觉语言模型（VLMs），但忽略了视觉表征本身可能存在的分布偏移。本文提出测试时噪声调优（TNT），首次在视觉输入空间中直接优化可学习噪声，实现仅凭单个测试样本即可自适应地学习特征表示。同时引入视图间表征对齐策略，显式约束嵌入距离的一致性，确保多视角下特征表示稳定。结合缩放逻辑斯蒂和置信度高的视图选择，TNT显著提升了VLM的泛化能力和校准效果，在自然分布基准上平均提升7.38%，跨数据集评估提升0.80%，超越零样本CLIP。该方法为处理分布外数据提供了坚实基础。
原文摘要 · Abstract (English)
Recently, test-time adaptation has garnered attention as a method for tuning models without labeled data. The conventional modus operandi for adapting pre-trained vision-language models (VLMs) during test-time primarily focuses on tuning learnable prompts; however, this approach overlooks potential distribution shifts in the visual representations themselves. In this work, we address this limitation by introducing Test-Time Noise Tuning (TNT), a novel method for handling unpredictable shifts in the visual space. TNT leverages, for the first time, a noise adaptation strategy that optimizes learnable noise directly in the visual input space, enabling adaptive feature learning from a single test sample. We further introduce a novel approach for inter-view representation alignment by explicitly enforcing coherence in embedding distances, ensuring consistent feature representations across views. Combined with scaled logits and confident view selection at inference, TNT substantially enhances VLM generalization and calibration, achieving average gains of +7.38% on natural distributions benchmark and +0.80% on cross-dataset evaluations over zero-shot CLIP. These improvements lay a strong foundation for adaptive out-of-distribution handling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。