通过多视角生成增强视觉推理,无需微调模型即可提升准确率。
MPCAR: Multi-Perspective Contextual Augmentation for Enhanced Visual Reasoning in Large Vision-Language Models
- 用模型生成多个视角描述,融合成更丰富的推理提示
- 在GQA、VQA-CP v2等数据集上显著提升准确率
- 适合需要深度理解视觉细节的复杂问答任务
尽管大型视觉语言模型(LVLMs)取得进展,但在需深层上下文理解、多角度分析或精细细节识别的复杂视觉推理任务中仍面临挑战。现有方法多依赖单次图像编码和提示,难以充分捕捉细微视觉信息。受“策略性生成额外信息可作为有益上下文增强”启发,我们提出面向推理的多视角上下文增强(MPCAR),一种无需微调模型参数的推理时策略。MPCAR分三步:首先,LVLM从不同角度生成N个多样化且互补的描述或初步推理路径;其次,将这些描述与原始问题智能融合,构建全面增强的提示;最后,利用该丰富提示引导LVLM进行深度推理并生成最终答案。在GQA、VQA-CP v2和ScienceQA(Image-VQA)等挑战性视觉问答数据集上的实验证明,MPCAR持续优于基线方法。定量结果显示,尤其在要求强上下文理解的任务中,准确率显著提升,人工评估也证实答案更具连贯性和完整性。消融实验进一步强调了多样提示模板和生成视角数量的重要性。本工作凸显了利用LVLM固有生成能力丰富输入上下文的有效性,从而释放其在复杂多模态任务中的潜在推理能力。
原文摘要 · Abstract (English)
Despite significant advancements, Large Vision-Language Models (LVLMs) continue to face challenges in complex visual reasoning tasks that demand deep contextual understanding, multi-angle analysis, or meticulous detail recognition. Existing approaches often rely on single-shot image encoding and prompts, limiting their ability to fully capture nuanced visual information. Inspired by the notion that strategically generated "additional" information can serve as beneficial contextual augmentation, we propose Multi-Perspective Contextual Augmentation for Reasoning (MPCAR), a novel inference-time strategy designed to enhance LVLM performance. MPCAR operates in three stages: first, an LVLM generates N diverse and complementary descriptions or preliminary reasoning paths from various angles; second, these descriptions are intelligently integrated with the original question to construct a comprehensive context-augmented prompt; and finally, this enriched prompt guides the ultimate LVLM for deep reasoning and final answer generation. Crucially, MPCAR achieves these enhancements without requiring any fine-tuning of the underlying LVLM's parameters. Extensive experiments on challenging Visual Question Answering (VQA) datasets, including GQA, VQA-CP v2, and ScienceQA (Image-VQA), demonstrate that MPCAR consistently outperforms established baseline methods. Our quantitative results show significant accuracy gains, particularly on tasks requiring robust contextual understanding, while human evaluations confirm improved coherence and completeness of the generated answers. Ablation studies further highlight the importance of diverse prompt templates and the number of generated perspectives. This work underscores the efficacy of leveraging LVLMs' inherent generative capabilities to enrich input contexts, thereby unlocking their latent reasoning potential for complex multimodal tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。