构建细粒度模糊视觉问答数据集,让模型学会根据模糊程度选择回应策略。
AQuA: Toward Strategic Response Generation for Ambiguous Visual Questions
- 按模糊程度分四类,标注每类最优回应策略。
- 多数模型盲目自信回答,无法识别模糊性。
- 微调后模型可自适应选答、追问或列选项,表现超越基线。
视觉问答(VQA)是评估视觉语言模型(VLMs)能力的核心任务。现有基准多为清晰无歧义的图文问答对,而真实场景常存在不同程度的模糊性,需细致推理与恰当响应策略。尽管近期研究开始关注模糊性,但仍缺乏(1)模糊性的系统分类和(2)支持策略感知响应的数据集与模型。本文提出模糊视觉问答(AQuA),一个细粒度数据集,将模糊VQA实例按性质与程度分为四类,并标注每类最优响应策略。对多种开源与专有VLM的评估显示,多数模型未能根据模糊类型调整策略,常给出过度自信的答案。为此,我们在AQuA上微调VLM,使其能自适应选择多种策略:直接回答、基于上下文推断意图、列出可能答案或请求澄清。在AQuA上训练的VLM实现战略化响应,具备识别模糊、管理不确定性及生成上下文适配响应的能力,性能优于开源与闭源基线。
原文摘要 · Abstract (English)
Visual Question Answering (VQA) is a core task for evaluating the capabilities of Vision-Language Models (VLMs). Existing VQA benchmarks primarily feature clear and unambiguous image-question pairs, whereas real-world scenarios often involve varying degrees of ambiguity that require nuanced reasoning and context-appropriate response strategies. Although recent studies have begun to address ambiguity in VQA, they lack (1) a systematic categorization of ambiguity levels and (2) datasets and models that support strategy-aware responses. In this paper, we introduce Ambiguous Visual Question Answering (AQuA), a fine-grained dataset that classifies ambiguous VQA instances into four levels according to the nature and degree of ambiguity, along with the optimal response strategy for each case. Our evaluation of diverse open-source and proprietary VLMs shows that most models fail to adapt their strategy to the ambiguity type, frequently producing overconfident answers rather than seeking clarification or acknowledging uncertainty. To address this challenge, we fine-tune VLMs on AQuA, enabling them to adaptively choose among multiple response strategies, such as directly answering, inferring intent from contextual cues, listing plausible alternatives, or requesting clarification. VLMs trained on AQuA achieve strategic response generation for ambiguous VQA, demonstrating the ability to recognize ambiguity, manage uncertainty, and respond with context-appropriate strategies, while outperforming both open-source and closed-source baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。