测试视觉语言模型是否依赖数据先验而非真实视觉推理
Probing Visual Language Priors in VLMs
- 设计新基准ViLP，用生成图像和异常问答对检测模型依赖
- GPT-4在该基准上仅66.17%准确率，远低于人类表现
- 提出自提升框架，通过图像噪声扰动增强模型视觉关注
尽管视觉语言模型（VLMs）取得进展，它们可能过度依赖训练数据中的视觉语言先验，而非真正的视觉推理。为此，我们引入了ViLP基准，该基准包含通过图像生成模型合成的分布外图像和分布外问答对。每个问题配有三个可能的答案和三张对应图像：一张仅靠文本先验即可解答，两张需要实际视觉推理。人类接近满分表现，而现代VLMs表现不佳，例如GPT-4在ViLP上仅达到66.17%准确率。为缓解此问题，我们提出一种自提升框架，让模型生成新的视觉问答数据，并施加像素级和语义级扰动，形成“好-坏”图像对用于自训练。训练目标促使模型更关注真实视觉输入，实验证明该方法能有效提升开源VLMs（如LLaVA-v1.5和Cambrian）性能。
原文摘要 · Abstract (English)
Despite recent advances in Vision-Language Models (VLMs), they may over-rely on visual language priors existing in their training data rather than true visual reasoning. To investigate this, we introduce ViLP, a benchmark featuring deliberately out-of-distribution images synthesized via image generation models and out-of-distribution Q&A pairs. Each question in ViLP is coupled with three potential answers and three corresponding images: one that can be resolved by text priors alone and two that demand visual reasoning. Although, humans achieve near-perfect accuracy, modern VLMs falter; for instance, GPT-4 achieves only 66.17% on ViLP. To alleviate this, we propose a self-improving framework in which models generate new VQA data, then apply pixel-level and semantic corruptions to form "good-bad" image pairs for self-training. Our training objectives compel VLMs to focus more on the actual visual inputs, and we demonstrate their effectiveness in boosting the performance of open-source VLMs, including LLaVA-v1.5 and Cambrian.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。