让视觉语言模型学会看懂图像基本几何信息，提升基础视觉理解能力。
Are VLMs Really Blind
- 用问题关键词生成聚焦图像细节的描述，引导模型精准回答。
- 在几何推理任务上显著优于直接VQA，无需额外微调。
- 适合希望提升模型基础视觉理解的开发者与研究者。
视觉语言模型在处理复杂任务如光学字符识别（OCR）、视觉问答（VQA）和高级几何推理方面表现优异，但在低层基础视觉任务上表现不佳，而这些任务对人类来说极为简单。本文旨在探究这些模型是否真的对几何推理‘视而不见’，以及是否存在提升其能力的方法。为此，我们提出一种自动化的数据提取管道：根据具体问题提取关键词，生成聚焦图像关键信息的描述性语句；该语句作为上下文输入语言模型，从而提供精确答案。整个过程不依赖外部微调，仅通过提示工程实现性能提升，显著增强了模型在基础视觉理解上的表现。
原文摘要 · Abstract (English)
Vision Language Models excel in handling a wide range of complex tasks, including Optical Character Recognition (OCR), Visual Question Answering (VQA), and advanced geometric reasoning. However, these models fail to perform well on low-level basic visual tasks which are especially easy for humans. Our goal in this work was to determine if these models are truly "blind" to geometric reasoning or if there are ways to enhance their capabilities in this area. Our work presents a novel automatic pipeline designed to extract key information from images in response to specific questions. Instead of just relying on direct VQA, we use question-derived keywords to create a caption that highlights important details in the image related to the question. This caption is then used by a language model to provide a precise answer to the question without requiring external fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。