让模型从图片中直接读题作答,更贴近真实视觉理解场景。
VoQA: Visual-only Question Answering
- 题目和场景同在一张图中,模型仅靠视觉推理。
- 现有模型在纯视觉下性能显著下降,差距超30%。
- 通过视觉对齐微调,提升模型看图答题能力。
视觉理解需同时解析自然场景与其中的文本信息,推动了视觉问答(VQA)任务的发展。然而,当前的VQA基准忽视了问题嵌入图像的场景,而高级智能体应能像人类一样直接从视觉中获取问题。为此,我们提出视觉仅问答(VoQA),即问题与场景均出现在同一张图像中,要求模型仅通过视觉感知与推理完成任务。该设定更贴近真实视觉交互场景,适用于问题或指令直接嵌入视觉内容的场景。在纯视觉零样本、提示引导及OCR辅助三种设置下的评估显示,当前模型性能相比传统VQA明显下降。为应对这一挑战,我们研究了问题对齐微调策略,引导模型优先理解视觉问题再进行推理。结合VoQA数据集与这些策略,实现了稳健的纯视觉推理能力,并保持了对传统VQA任务的跨任务泛化性,体现了通过VoQA训练所增强的视觉与文本推理互补能力。代码与数据已公开。
原文摘要 · Abstract (English)
Visual understanding requires interpreting both natural scenes and the textual information that appears within them, motivating tasks such as Visual Question Answering (VQA). However, current VQA benchmarks overlook scenarios with visually embedded questions, whereas advanced agents should be able to see the question without separate text input as humans. We introduce Visual-only Question Answering (VoQA), where both the scene and the question appear within a single image, requiring models to perceive and reason purely through vision. This setting supports more realistic visual understanding and interaction in scenarios where questions or instructions are embedded directly in the visual scene. Evaluations under pure visual-only zero-shot, prompt-guided and OCR-assisted settings show that current models exhibit a clear performance drop compared to traditional VQA. To address this, we investigate question-alignment fine-tuning strategies designed to guide models toward interpreting the visual question prior to reasoning. Leveraging VoQA dataset together with these strategies yields robust vision-only reasoning while preserving cross-task generalization to traditional VQA, reflecting the complementary visual and textual reasoning capabilities fostered through VoQA training. The code and data are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。