让大模型更懂图片细节，减少答错的幻觉。
ReVA: A Region-Aware Visual Assistant for Visually Grounded Question Answering
- 用双桥结构融合图像与局部区域特征，增强视觉定位能力。
- 在POPE数据集上达82.85%准确率，优于基础模型1.71个百分点。
- 适合需要精准空间理解的视觉问答任务，如医疗影像分析。
多模态大语言模型（MLLM）在视觉问答（VQA）上取得显著进展，但仍难以处理需精确空间推理和细粒度视觉理解的问题，常出现对象、属性和空间幻觉。为解决此问题，我们提出ReVA，一种区域感知的VQA模型。该模型采用冻结的CLIP ViT-L/14视觉变换器与Qwen2.5-7B-Instruct大语言模型，通过双桥结构将整体图像与区域级表征对齐至语言模型嵌入空间。图像桥将最后一层Transformer特征映射为图像标记；区域桥则从ViT各层级的增强中间特征中裁剪出区域特征，提取早期纹理与后期物体线索，生成每个边界框对应的K个区域标记。模型使用检测器堆栈自动提供零样本边界框，包括问题无关与问题相关的框，基于RAM++、spaCy和Grounding DINO。图像标记与区域标记拼接作为语言模型提示前缀，联合编码场景上下文与细粒度区域证据。在VQAv2、MMBench、POPE和SEED-Bench上评估，ReVA在POPE上实现82.85%的平均F1，较仅用图像标记的基线（81.14%）提升1.71个百分点。结果表明，显式的区域感知视觉表示可有效减少对象幻觉，提升模型事实性与视觉对齐能力。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have achieved remarkable progress in Visual Question Answering (VQA), yet they continue to struggle with questions requiring precise spatial reasoning and fine-grained visual understanding. These limitations often manifest as object, attribute, and spatial hallucinations, where models generate confident but visually unsupported responses due to insufficient region-level and fine-grained visual grounding. To address this challenge, we propose ReVA, a region-aware VQA model that employs a frozen CLIP ViT-L/14 Vision Transformer (ViT) and a Qwen2.5-7B-Instruct large language model (LLM) connected through a dual bridge that aligns both whole-image and region-level representations with the LLM's embedding space. The image bridge maps final transformer block features into image tokens. The region bridge maps cropped features from enriched intermediate features across ViT blocks so early texture and later object cues are more evident, into K region tokens for every bounding box. ReVA uses a detector stack that supplies automatic zero-shot bounding boxes that are both question-agnostic and question-dependent, using RAM++ (Recognize Anything Model), spaCy, and Grounding DINO. The image tokens and region tokens are concatenated as an LLM prompt prefix to jointly encode scene-level context and fine-grained regional evidence when answering questions. Evaluated on VQAv2, MMBench, POPE, and SEED-Bench, ReVA achieves 82.85% mean F1 on POPE, compared with 81.14% for an image-token baseline without region tokens. These results demonstrate that explicit region-aware visual representations reduce object hallucination and improve the factual grounding of MLLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。