通过区域验证链减少大模型幻觉，提升视觉理解准确性。
R-CoV: Region-Aware Chain-of-Verification for Alleviating Object Hallucinations in LVLMs
- 让模型分步检查图像特定区域，自我验证对象是否存在。
- 在多个基准上使幻觉率降低超过30%，效果显著。
- 无需训练、不依赖外部工具，适合各类视觉语言模型使用。
大型视觉语言模型（LVLMs）在多模态理解与推理任务中表现优异，但仍存在对象幻觉问题，即声称图像中不存在的对象。为解决此问题，本文提出区域感知的验证链方法（R-CoV），一种后处理式视觉验证机制。受人类关注图像局部细节的启发，R-CoV从模型自身提取区域级信息作为验证线索，包含六个步骤：初始响应生成、实体提取、坐标生成、区域描述、验证执行与最终响应生成。该方法无需训练、不依赖外部检测模型，可无缝集成至多种LVLM中。在多个主流幻觉评测基准上的实验表明，R-CoV能显著降低对象幻觉，提升模型可靠性。
原文摘要 · Abstract (English)
Large vision-language models (LVLMs) have demonstrated impressive performance in various multimodal understanding and reasoning tasks. However, they still struggle with object hallucinations, i.e., the claim of nonexistent objects in the visual input. To address this challenge, we propose Region-aware Chain-of-Verification (R-CoV), a visual chain-of-verification method to alleviate object hallucinations in LVLMs in a post-hoc manner. Motivated by how humans comprehend intricate visual information -- often focusing on specific image regions or details within a given sample -- we elicit such region-level processing from LVLMs themselves and use it as a chaining cue to detect and alleviate their own object hallucinations. Specifically, our R-CoV consists of six steps: initial response generation, entity extraction, coordinate generation, region description, verification execution, and final response generation. As a simple yet effective method, R-CoV can be seamlessly integrated into various LVLMs in a training-free manner and without relying on external detection models. Extensive experiments on several widely used hallucination benchmarks across multiple LVLMs demonstrate that R-CoV can significantly alleviate object hallucinations in LVLMs. Project page: https://github.com/Jiahao000/R-CoV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。