提出MVP方法，让视觉语言模型更抗提示词变化干扰。
Modeling Variants of Prompts for Vision-Language Models
- 将提示词拆解为模板与类别名，用VAE建模结构分布。
- 在11个数据集上验证，对提示变化的鲁棒性显著提升。
- 适合需要稳定提示输入的应用场景，如开放域识别。
大型预训练视觉语言模型（VLMs）通过人类语言增强下游任务表现，但如CLIP等模型对提示模板设计极为敏感。尽管提示学习可通过可学习提示缓解此问题，但其结果难以被人类理解。本文构建了RobustPrompt基准，包含数百个精心设计的提示模板，分为六类，覆盖常见表达形式。同时提出MVP方法，通过将提示分解为模板与类别名，并利用变分自编码器（VAE）建模多样提示结构分布，从而降低模型对提示变化的敏感性。在11个数据集上的实验表明，MVP在不降低性能的前提下，显著提升了模型对输入提示变化的鲁棒性。代码已开源。
原文摘要 · Abstract (English)
Large pre-trained vision-language models (VLMs) offer a promising approach to leveraging human language for enhancing downstream tasks. However, VLMs such as CLIP face significant limitation: its performance is highly sensitive to prompt template design. Although prompt learning methods can address the sensitivity issue by replacing natural language prompts with learnable ones, they are incomprehensible to humans. Ensuring consistent performance across various prompt templates enables models to adapt seamlessly to diverse phrasings, enhancing their ability to handle downstream tasks without requiring extensive prompt engineering. In this work, we introduce the RobustPrompt Benchmark, a systematic benchmark to evaluate robustness to different prompt templates for VLMs. It includes a dataset with hundreds of carefully designed prompt templates, divided into six types, covering a wide variety of commonly used templates. Beside the benchmark, we propose Modeling Variants of Prompts (MVP), a simple yet effective method that mitigates sensitivity by modeling variants of prompt structures. The innovation of MVP lies in decoupling prompts into templates and class names, and using Variational Autoencoders (VAE) to model the distribution of diverse prompt structures. Experiments across 11 datasets demonstrate that MVP can greatly enhance model robustness to variations in input prompts without a drop in performance. The code is available at https://github.com/liaolea/MVP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。