发现视觉问答中的简单选项偏差，提出工具自动生成难负例以消除偏差。
Mitigating Easy Option Bias in Multiple-Choice Question Answering
- 通过视觉-选项相似性匹配发现模型可跳过问题直接选答案
- 在新标注数据上，模型在无问题条件下准确率降至随机水平
- 适合评估视觉语言模型真实问答能力的研究者使用
本研究首次观察到多个视觉问答基准（如MMStar、RealWorldQA、SEED-Bench、Next-QA、STAR和Video-MME）存在简单选项偏差（EOB）。该偏差使视觉语言模型（VLM）仅凭视觉输入（V）和选项（O）即可正确作答，无需使用问题（Q）。通过定位实验，我们发现偏差源于视觉相关性的不平衡：正确答案在特征空间中与视觉内容更接近，导致模型可通过简单匹配选出答案。为解决此问题，我们提出GroundAttack工具包，可自动生成在视觉上与正确答案同样合理的难负例。我们在NExT-QA和MMStar数据集上应用该工具，构建了新的无EOB标注数据。在这些新数据上，当前VLM在仅用（V+O）的设置下准确率接近随机水平，在（V+Q+O）设置下也降至非饱和状态，从而提供更真实的模型问答能力评估。代码与新标注数据将尽快公开。
原文摘要 · Abstract (English)
In this early study, we observe an Easy-Options Bias (EOB) issue in some multiple-choice Visual Question Answering (VQA) benchmarks such as MMStar, RealWorldQA, SEED-Bench, Next-QA, STAR benchmark and Video-MME. This bias allows vision-language models (VLMs) to select the correct answer using only the vision (V) and options (O) as inputs, without the need for the question (Q). Through grounding experiments, we attribute the bias to an imbalance in visual relevance: the correct answer typically aligns more closely with the visual contents than the negative options in feature space, creating a shortcut for VLMs to infer the answer via simply vision-option similarity matching. To fix this, we introduce GroundAttack, a toolkit that automatically generates hard negative options as visually plausible as the correct answer. We apply it to the NExT-QA and MMStar datasets, creating new EOB-free annotations. On these EOB-free annotations, current VLMs approach to random accuracies under (V+O) settings, and drop to non-saturated accuracies under (V+Q+O) settings, providing a more realistic evaluation of VLMs' QA ability. Codes and new annotations will be released soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。