arXiv:2505.21844cs.CV2025-05NeurIPS被引 12

为视觉语言模型设计测试时自适应方法,提升开放词汇语义分割性能。

Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation

  • 基于多层级特征与多提示模板的熵最小化策略,适配分割任务。
  • 在87种测试场景中显著优于分类用的基线方法,单样本即可生效。
  • 无需额外数据或标签,可无缝集成到任意分割网络中。

最近,测试时自适应(TTA)在视觉语言模型图像分类中受到广泛关注,但在密集预测任务如开放词汇语义分割(OVSS)中仍被忽视。为此,我们提出一种专为分割任务设计的新型TTA方法——多层级多提示(MLMP)熵最小化。该方法融合中间视觉编码器层的特征,并在全局CLS token和局部像素级同时使用不同文本提示模板。该方法可作为即插即用模块应用于任意分割网络,无需额外训练数据或标签,在仅一个测试样本的情况下仍有效。此外,我们构建了一个全面的OVSS TTA基准套件,包含严格评估协议、9个分割数据集、15种常见合成噪声以及真实与渲染域偏移,共计87种不同测试场景,为未来开放词汇分割中的TTA研究提供标准化测试平台。实验表明,我们的方法在该套件上持续显著优于直接应用分类领域基线的TTA方法。代码与数据已公开于https://github.com/dosowiechi/MLMP。

原文摘要 · Abstract (English)

Recently, test-time adaptation has attracted wide interest in the context of vision-language models for image classification. However, to the best of our knowledge, the problem is completely overlooked in dense prediction tasks such as Open-Vocabulary Semantic Segmentation (OVSS). In response, we propose a novel TTA method tailored to adapting VLMs for segmentation during test time. Unlike TTA methods for image classification, our Multi-Level and Multi-Prompt (MLMP) entropy minimization integrates features from intermediate vision-encoder layers and is performed with different text-prompt templates at both the global CLS token and local pixel-wise levels. Our approach could be used as plug-and-play for any segmentation network, does not require additional training data or labels, and remains effective even with a single test sample. Furthermore, we introduce a comprehensive OVSS TTA benchmark suite, which integrates a rigorous evaluation protocol, nine segmentation datasets, 15 common synthetic corruptions, and additional real and rendered domain shifts, \textbf{with a total of 87 distinct test scenarios}, establishing a standardized and comprehensive testbed for future TTA research in open-vocabulary segmentation. Our experiments on this suite demonstrate that our segmentation-tailored method consistently delivers significant gains over direct adoption of TTA classification baselines. Code and data are available at https://github.com/dosowiechi/MLMP.

视觉语言模型测试时自适应语义分割开放词汇

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。