通过视觉与文本双重锚定，减少大模型幻觉并提升细粒度理解。
PostAlign: Multimodal Grounding as a Corrective Lens for MLLMs
- 引入双路径校准：视觉定位目标，文本生成推理依据。
- 负例拒绝机制有效区分真实与虚假对象，降低幻觉率。
- 适配复杂查询的推理策略调整，适合高精度视觉问答场景。
多模态大语言模型（MLLMs）在图像描述、视觉问答等任务中表现优异，但常因语言先验导致过度依赖虚假关联，忽视真实视觉信息。为此，我们提出MMGrounded-PostAlign框架，通过多模态对齐增强视觉理解并抑制幻觉。该框架包含两个核心模块：视觉定位模块可识别图像中的指代对象，文本生成模块则构建答案推理链条，确保输出同时基于视觉和语言证据。为减少幻觉，视觉模块引入负例拒绝机制，区分由语言偏见诱导的不存在对象；文本模块则采用选择性推理机制，根据问题复杂度动态调整推理策略。在POPE、HaloQuest、VQAv2、MME和MMBench等多个基准上进行评估，结果显示模型在细粒度视觉理解与幻觉抑制方面均有显著提升。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) excel in vision-language tasks, such as image captioning and visual question answering. However, they often suffer from over-reliance on spurious correlations, primarily due to linguistic priors that distract the model from leveraging actual visual information. To address these issues, we introduce MMGrounded-PostAlign, a post-multimodal alignment framework designed to enhance the visual understanding capabilities and mitigate the hallucinations of MLLMs. Our framework incorporates a multimodal grounding module for both visual grounding, which identifies the referred object in the image, and textual grounding, which generates the rationale for the final answer, ensuring that outputs are anchored in both visual and textual evidence. To mitigate the hallucinations, we introduce a negative rejection mechanism in the visual grounding module to distinguish grounded entities from non-existent objects influenced by linguistic biases. On the textual grounding side, we propose a selective reasoning mechanism that adjusts the model's reasoning strategy based on query complexity. Extensive evaluations are conducted on benchmarks such as POPE, HaloQuest, VQAv2, MME, and MMBench showing significant improvements in fine-grained visual understanding and hallucination suppression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。