arXiv:2412.07518cs.CV2024-12被引 6

提出HCOENet框架,减少视觉语言模型在交通场景中的幻觉并增强关键物体描述。

Hallucination Elimination and Semantic Enhancement Framework for Vision-Language Models in Traffic Scenarios

  • 通过交叉验证机制过滤实体,从图像直接提取关键物体进行纠正
  • 在POPE基准上使Mini-InternVL-4B模型F1提升12.58%,mPLUG-Owl3提升4.28%
  • 适合自动驾驶系统中需高可靠性的多模态理解任务

大型视觉语言模型(LVLMs)在多模态理解和生成任务中表现出色,但常产生看似合理却与图像不符的幻觉性文本,可能导致自动驾驶系统误判。为此,本文提出HCOENet——一种即插即用的思维链修正方法,用于消除物体幻觉并增强初始回答中遗漏的关键物体描述。HCOENet采用交叉验证机制过滤实体,并直接从图像中提取关键物体以丰富描述内容。在POPE基准上的实验表明,HCOENet使Mini-InternVL-4B和mPLUG-Owl3模型的F1分数分别提升12.58%和4.28%。基于开放校园场景采集的图像进行的定性分析进一步验证了该方法的实际适用性。相比GPT-4o模型,HCOENet实现相当的描述性能,同时显著降低使用成本。最后,本文构建了两个面向交通场景的新语义理解数据集:CODA_desc和nuScenes_desc,以支持后续研究。代码与数据集已开源于https://github.com/fjq-tongji/HCOENet。

原文摘要 · Abstract (English)

Large vision-language models (LVLMs) have demonstrated remarkable capabilities in multimodal understanding and generation tasks. However, these models occasionally generate hallucinatory texts, resulting in descriptions that seem reasonable but do not correspond to the image. This phenomenon can lead to wrong driving decisions of the autonomous driving system. To address this challenge, this paper proposes HCOENet, a plug-and-play chain-of-thought correction method designed to eliminate object hallucinations and generate enhanced descriptions for critical objects overlooked in the initial response. Specifically, HCOENet employs a cross-checking mechanism to filter entities and directly extracts critical objects from the given image, enriching the descriptive text. Experimental results on the POPE benchmark demonstrate that HCOENet improves the F1-score of the Mini-InternVL-4B and mPLUG-Owl3 models by 12.58% and 4.28%, respectively. Additionally, qualitative results using images collected in open campus scene further highlight the practical applicability of the proposed method. Compared with the GPT-4o model, HCOENet achieves comparable descriptive performance while significantly reducing costs. Finally, two novel semantic understanding datasets, CODA_desc and nuScenes_desc, are created for traffic scenarios to support future research. The codes and datasets are publicly available at https://github.com/fjq-tongji/HCOENet.

视觉语言模型幻觉消除自动驾驶多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。