让AI通过自问自答理解图像，提升多步推理准确率。
Instruction-tuned Self-Questioning Framework for Multimodal Reasoning
- 分三模块生成图文相关的子问题与答案，逐步推理主问题。
- 在VQA任务上比现有方法更准，显著提升多步推理性能。
- 可解释性强，适合需要透明推理过程的研究与应用。
近年来，得益于大语言模型（LLMs）的发展，视觉-语言理解领域取得显著进展。然而，对于需要多步推理的简单问题，现有方法仍表现不足。近期研究尝试通过迭代生成子问题和答案来解决该问题，但存在两大缺陷：1）使用无法读取视觉信息的黑盒LLMs时，图像的细粒度内容无法被利用；2）内部机制不可见，难以复现。为此，本文提出SQ-InstructBLIP框架，通过迭代生成图像感知的子问题与子答案，提升推理性能。该框架包含共享架构的Questioner、Answerer与Reasoner：Questioner与Answerer生成辅助子问题与答案，Reasoner则结合这些信息对主问题进行推理。实验表明，使用生成的子问题作为额外信息时，该方法在视觉问答（VQA）任务上的推理准确性优于先前工作。
原文摘要 · Abstract (English)
The field of vision-language understanding has been actively researched in recent years, thanks to the development of Large Language Models~(LLMs). However, it still needs help with problems requiring multi-step reasoning, even for very simple questions. Recent studies adopt LLMs to tackle this problem by iteratively generating sub-questions and answers. However, there are disadvantages such as 1) the fine-grained visual contents of images are not available using LLMs that cannot read visual information, 2) internal mechanisms are inaccessible and difficult to reproduce by using black-box LLMs. To solve these problems, we propose the SQ (Self-Questioning)-InstructBLIP, which improves inference performance by generating image-aware informative sub-questions and sub-answers iteratively. The SQ-InstructBLIP, which consists of a Questioner, Answerer, and Reasoner that share the same architecture. Questioner and Answerer generate sub-questions and sub-answers to help infer the main-question, and Reasoner performs reasoning on the main-question considering the generated sub-question information. Our experiments show that the proposed method SQ-InstructBLIP, which uses the generated sub-questions as additional information when solving the VQA task, performs more accurate reasoning than the previous works.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。