提出PARC框架，量化视觉语言模型对提示的敏感性。
PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Models
- 构建多模态提示变化集，分析文本与图像提示的扰动影响。
- 22个模型中InternVL2系列最抗提示变化，错误率仅12.3%。
- 发现提示敏感性与训练数据分布相关，适合模型评估与改进。
视觉语言模型（VLMs）接收用户设计的文本提示和视觉输入，广泛应用于各类实际问题。由于大型语言模型（LLMs）对提示高度敏感，因此需明确VLMs是否继承这一不稳定性。本文提出PARC（Prompt Analysis via Reliability and Calibration），一个基于三个支柱的提示敏感性分析框架：(1) 在语言与视觉域中构建合理的提示变化；(2) 提出具有内在保证的新模型可靠性评分；(3) 引入校准步骤，实现跨数据集与提示范围的分析。实验表明，VLMs在视觉域中重现了LLMs的语言提示敏感性，最具破坏性的变化会改变预期答案。在22个被评估的模型中，InternVL2系列表现出显著鲁棒性，错误率最低为12.3%。此外，提示敏感性可能与训练数据分布有关。代码已开源于https://github.com/NVlabs/PARC。
原文摘要 · Abstract (English)
Vision language models (VLMs) respond to user-crafted text prompts and visual inputs, and are applied to numerous real-world problems. VLMs integrate visual modalities with large language models (LLMs), which are well known to be prompt-sensitive. Hence, it is crucial to determine whether VLMs inherit this instability to varying prompts. We therefore investigate which prompt variations VLMs are most sensitive to and which VLMs are most agnostic to prompt variations. To this end, we introduce PARC (Prompt Analysis via Reliability and Calibration), a VLM prompt sensitivity analysis framework built on three pillars: (1) plausible prompt variations in both the language and vision domain, (2) a novel model reliability score with built-in guarantees, and (3) a calibration step that enables dataset- and prompt-spanning prompt variation analysis. Regarding prompt variations, PARC's evaluation shows that VLMs mirror LLM language prompt sensitivity in the vision domain, and most destructive variations change the expected answer. Regarding models, outstandingly robust VLMs among 22 evaluated models come from the InternVL2 family. We further find indications that prompt sensitivity is linked to training data. The code will be at https://github.com/NVlabs/PARC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。