让视觉问答模型学会自己提问，提升复杂推理能力。
Elevating Visual Question Answering through Implicitly Learned Reasoning Pathways in LVLMs
- 通过端到端训练让模型自动生成中间问题并回答
- 在ScienceQA和VQAv2上超越现有最优模型
- 适合需要强逻辑推理的多模态任务研究者
大型视觉语言模型（LVLMs）在多模态任务中表现突出，但在需要多步推理的复杂视觉理解任务中仍存在不足。为此，我们提出MF-SQ-LLaVA，通过端到端训练使模型实现隐式自提问。方法在视觉问答数据集中引入由子问题与答案组成的推理链，并采用多任务损失训练模型生成中间步骤并预测最终答案。在ScienceQA和VQAv2数据集上进行大量实验表明，MF-SQ-LLaVA显著优于现有最先进模型，包括基础版LLaVA和原始SQ-LLaVA。消融实验证明了各组件的有效性，人工评估也证实该方法提升了推理过程的准确性和连贯性。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) have shown remarkable progress in various multimodal tasks, yet they often struggle with complex visual reasoning that requires multi-step inference. To address this limitation, we propose MF-SQ-LLaVA, a novel approach that enhances LVLMs by enabling implicit self-questioning through end-to-end training. Our method involves augmenting visual question answering datasets with reasoning chains consisting of sub-question and answer pairs, and training the LVLM with a multi-task loss that encourages the generation and answering of these intermediate steps, as well as the prediction of the final answer. We conduct extensive experiments on the ScienceQA and VQAv2 datasets, demonstrating that MF-SQ-LLaVA significantly outperforms existing state-of-the-art models, including the base LLaVA and the original SQ-LLaVA. Ablation studies further validate the contribution of each component of our approach, and human evaluation confirms the improved accuracy and coherence of the reasoning process enabled by our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。