发现多模态大模型答简单问题反而容易错，称其为‘模型懒惰’。
Difficult Task Yes but Simple Task No: Unveiling the Laziness in Multimodal LLMs
- 构建新基准LazyBench，对比同一图像的复杂与简单问题回答表现
- 发现强模型在简单问题上错误率更高，约一半现有数据集失败由懒惰导致
- 链式思维提示可有效缓解懒惰，适合关注模型可靠性研究者
多模态大语言模型（MLLMs）虽能处理复杂任务，却常在简单视觉问答（VQA）中出错。本文揭示这一现象背后的行为差异：模型在回答关于图像的简单问题（如是/否问题）时更易出错，尽管能正确描述图像，称为“模型懒惰”。为此，我们手工构建了涵盖是/否、选择题、简答和图像描述的LazyBench基准，包含同一图像主题的多种任务。基于该基准，发现当前主流模型（如GPT-4o、Gemini-1.5-pro、Claude 3、LLaVA-v1.5-13B）普遍存在懒惰现象，且更强模型表现更严重。分析VQA v2（LLaVA-v1.5-13B）发现，约一半失败案例源于懒惰，凸显充分调用模型能力的重要性。初步探索表明，链式思维（CoT）提示可有效缓解该问题。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) demonstrate a strong understanding of the real world and can even handle complex tasks. However, they still fail on some straightforward visual question-answering (VQA) problems. This paper dives deeper into this issue, revealing that models tend to err when answering easy questions (e.g. Yes/No questions) about an image, even though they can correctly describe it. We refer to this model behavior discrepancy between difficult and simple questions as model laziness. To systematically investigate model laziness, we manually construct LazyBench, a benchmark that includes Yes/No, multiple choice, short answer questions, and image description tasks that are related to the same subjects in the images. Based on LazyBench, we observe that laziness widely exists in current advanced MLLMs (e.g. GPT-4o, Gemini-1.5-pro, Claude 3 and LLaVA-v1.5-13B), and it is more pronounced on stronger models. We also analyze the VQA v2 (LLaVA-v1.5-13B) benchmark and find that about half of its failure cases are caused by model laziness, which further highlights the importance of ensuring that the model fully utilizes its capability. To this end, we conduct preliminary exploration on how to mitigate laziness and find that chain of thought (CoT) can effectively address this issue.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。