提出轻量级方法，在不修改模型的情况下提升视觉语言模型的零样本泛化能力。
Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision-Language Models
- 从文本嵌入中提取语义主方向，通过少量参数调节隐空间表示。
- 在多个数据集上性能超越或媲美现有方法，推理速度提升8倍，内存减少12倍。
- 适合需要快速部署、低资源场景的零样本视觉任务应用。
视觉语言模型在零样本推理中表现优异，但在测试时遭遇领域偏移时性能下降。为此，近期出现了基于事件的测试时自适应策略，可对单个无标签图像进行适配。然而，现有方法如测试时提示调优通常需反向传播大量编码器权重或修改核心模型组件。本文提出谱感知测试时导向（STS），一种轻量级自适应框架：从文本嵌入中提取谱子空间以定义主语义方向，并通过少量每样本偏移参数，以最小化多视图下熵的方式，实现谱感知的隐空间导向。该方法完全在推理阶段操作于隐空间，无需对冻结编码器进行反向传播或修改。基于标准评估协议的全面实验表明，STS显著优于或媲美当前最先进测试时自适应方法，仅引入极少量额外参数，推理速度最高达8倍提升，内存占用降低12倍。代码已开源。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) excel at zero-shot inference but often degrade under test-time domain shifts. For this reason, episodic test-time adaptation strategies have recently emerged as powerful techniques for adapting VLMs to a single unlabeled image. However, existing adaptation strategies, such as test-time prompt tuning, typically require backpropagating through large encoder weights or altering core model components. In this work, we introduce Spectrum-Aware Test-Time Steering (STS), a lightweight adaptation framework that extracts a spectral subspace from the textual embeddings to define principal semantic directions and learns to steer latent representations in a spectrum-aware manner by adapting a small number of per-sample shift parameters to minimize entropy across augmented views. STS operates entirely at inference in the latent space, without backpropagation through or modification of the frozen encoders. Building on standard evaluation protocols, our comprehensive experiments demonstrate that STS largely surpasses or compares favorably against state-of-the-art test-time adaptation methods, while introducing only a handful of additional parameters and achieving inference speeds up to 8x faster with a 12x smaller memory footprint than conventional test-time prompt tuning. The code is available at https://github.com/kdafnis/STS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。