先思考再看图，用文本推理减少视觉幻觉
Thinking Before Looking: Improving Multimodal LLM Reasoning via Mitigating Visual Hallucination
- 先仅凭文本构建推理链，再引入图像避免误导
- 在多个视觉任务上零样本性能显著提升
- 适合需要高可靠性的多模态推理场景
多模态大语言模型（MLLM）已成视觉-语言任务的主流范式。现有链式思维（CoT）方法在适配MLLM时，因跨模态理解中幻觉风险升高而受限。本文发现，当前‘边看边想’的多模态CoT策略——即在观察图像的同时生成推理链——无法缓解由误导性图像引发的幻觉问题。为此，我们提出视觉推理链（VIC）框架：先仅基于文本上下文构建推理链，再引入视觉信息，有效降低跨模态偏差，提升多模态推理准确性。全面评估表明，VIC在多个视觉相关任务上显著改善零样本性能，减少幻觉并增强推理能力。代码已开源：https://github.com/Terry-Xu-666/visual_inference_chain。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have advanced the integration of visual and linguistic modalities, establishing themselves as the dominant paradigm for visual-language tasks. Current approaches like chain of thought (CoT) reasoning have augmented the cognitive capabilities of large language models (LLMs), yet their adaptation to MLLMs is hindered by heightened risks of hallucination in cross-modality comprehension. In this paper, we find that the thinking while looking paradigm in current multimodal CoT approaches--where reasoning chains are generated alongside visual input--fails to mitigate hallucinations caused by misleading images. To address these limitations, we propose the Visual Inference Chain (VIC) framework, a novel approach that constructs reasoning chains using textual context alone before introducing visual input, effectively reducing cross-modal biases and enhancing multimodal reasoning accuracy. Comprehensive evaluations demonstrate that VIC significantly improves zero-shot performance across various vision-related tasks, mitigating hallucinations while refining the reasoning capabilities of MLLMs. Our code repository can be found at https://github.com/Terry-Xu-666/visual_inference_chain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。