用细粒度图像对比测试MLLM对视觉概念的理解能力。
Detect, Describe, Discriminate: Moving Beyond VQA for MLLM Evaluation
- 通过自检索机制评估模型描述差异的能力
- 开源模型在细粒度区分上表现不如随机猜测
- 适合关注模型真实理解力的研究者
多选题视觉问答（VQA）虽能检验多模态大模型的视觉能力，但选择答案比生成答案更简单。本文提出新评测视角：通过模型能否唯一描述一对仅在特定视觉概念上有差异的图像来评估其对视觉概念的理解。我们构建了包含247对高度相似图像的D3基准，要求模型完成三步：检测差异、唯一描述目标图像、使其能从干扰项中被准确检索。基于自检索的白盒评估覆盖六种视觉模式，结果显示当前模型难以独立辨别细微视觉差异，开源模型表现甚至低于随机猜测。
原文摘要 · Abstract (English)
Visual Question Answering (VQA) with multiple choice questions enables a vision-centric evaluation of Multimodal Large Language Models (MLLMs). Although it reliably checks the existence of specific visual abilities, it is easier for the model to select an answer from multiple choices (VQA evaluation) than to generate the answer itself. In this work, we offer a novel perspective: we evaluate how well an MLLM understands a specific visual concept by its ability to uniquely describe two extremely similar images that differ only in the targeted visual concept. Specifically, we assess the ability of MLLMs to capture specific points of visual differences using self-retrieval, i.e., by retrieving the target image using its generated caption against the other image in the pair serving as the distractor. We curate 247 highly similar image pairs as part of the D3 benchmark. For each image pair, the model is prompted to: (1) Detect a specific visual difference, and (2) Describe the target image uniquely such that it (3) Discriminates the target image from the distractor. Self-retrieval within D3 enables whitebox evaluation across six different visual patterns, revealing that current models struggle to independently discern fine-grained visual differences, with open-source models failing to outperform random guess.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。