动态融合直觉与深思,提升视觉问答模型推理能力
Fast or Slow? Integrating Fast Intuition and Deliberate Thinking for Enhancing Visual Question Answering
- 根据问题复杂度切换快速直觉与深度分析策略
- 在四个数据集上显著提升开源与闭源模型表现
- 适合需要强视觉推理的多模态应用开发者
多模态大语言模型在视觉问答任务中仍面临复杂推理挑战。现有方法通过视觉提示标注所有检测到的物体,但这种方式会生成过多视觉标记,导致性能下降。本研究发现并非所有物体都同等重要,也非所有问题都需要视觉提示。基于双过程认知理论,提出FOCUS——一种即插即用的动态适配方法,结合快速直觉判断与严谨分析推理,增强模型的视觉-语言推理能力。对简单问题支持零样本高效推理;对复杂任务采用‘先概念后观察’策略聚焦关键视觉元素。在ScienceQA、TextQA、VizWiz和MME四个基准上的实验证明,FOCUS持续提升开源与闭源模型性能,消融实验进一步验证了多样化认知策略与精炼视觉信息结合的重要性。代码将公开。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) still struggle with complex reasoning tasks in Visual Question Answering (VQA). While current methods have advanced by incorporating visual prompts, our study uncovers critical limitations: these approaches indiscriminately annotate all detected objects for every visual question, generating excessive visual markers that degrade task performance. This issue stems primarily from a lack of focus on key visual elements, raising two important questions: Are all objects equally important, and do all questions require visual prompts? Motivated by Dual Process Theory, which distinguishes between instinctive and deliberate cognitive modes in human reasoning, we propose FOCUS, a plug-and-play approach that dynamically adapts to the complexity of questions, combining fast intuitive judgments with deliberate analytical reasoning to enhance the vision-language reasoning capability of the MLLM. For straightforward questions, FOCUS supports efficient zero-shot reasoning. For more complex tasks, it employs the conceptualizing before observation strategy to highlight critical elements. Extensive experiments on four benchmarks, ScienceQA, TextQA, VizWiz, and MME, demonstrate that FOCUS consistently improves the performance of both open-source and black-box MLLMs, achieving significant gains across all datasets. Ablation studies further validate the importance of combining diverse cognitive strategies with refined visual information for superior performance. Code will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。