给视觉语言模型的幻觉问题设计统一解决方案，按问题类型精准处理。
A Unified Hallucination Mitigation Framework for Large Vision-Language Models
- 先分类查询是感知还是推理，再针对性地减少幻觉。
- 在MMbench上提升13.44%~15.8%准确率，尤其在图像质量评估任务中。
- 适合需要高可信度图文回答的应用场景，如医疗或安全关键系统。
大型视觉语言模型（LVLM）在长文本生成中普遍存在幻觉问题，生成内容常与图像事实不符。现有方法多聚焦于推理过程或生成结果，但对不同类型的查询缺乏适配性。为此，本文提出统一框架Dentist，其核心是先对查询进行分类（感知类或推理类），再根据类别实施相应幻觉缓解策略，类比牙医先检查牙齿再制定治疗方案。该方法部署简单，在多个基准测试中表现优异：在MMbench的图像质量评估（Coarse Perception VQA）任务上，相比InstructBLIP/LLaVA/VisualGLM基线，准确率分别提升13.44%、10.2%和15.8%。
原文摘要 · Abstract (English)
Hallucination is a common problem for Large Vision-Language Models (LVLMs) with long generations which is difficult to eradicate. The generation with hallucinations is partially inconsistent with the image content. To mitigate hallucination, current studies either focus on the process of model inference or the results of model generation, but the solutions they design sometimes do not deal appropriately with various types of queries and the hallucinations of the generations about these queries. To accurately deal with various hallucinations, we present a unified framework, Dentist, for hallucination mitigation. The core step is to first classify the queries, then perform different processes of hallucination mitigation based on the classification result, just like a dentist first observes the teeth and then makes a plan. In a simple deployment, Dentist can classify queries as perception or reasoning and easily mitigate potential hallucinations in answers which has been demonstrated in our experiments. On MMbench, we achieve a 13.44%/10.2%/15.8% improvement in accuracy on Image Quality, a Coarse Perception visual question answering (VQA) task, over the baseline InstructBLIP/LLaVA/VisualGLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。