通过外部知识验证，让视频语言模型更少说谎。
ResNetVLLM-2: Addressing ResNetVLLM's Multi-Modal Hallucinations
- 用改进的Lynx模型检测图文语义是否对得上
- 推理时动态构建知识库，用RAG减少错误生成
- 在ActivityNet-QA上准确率从54.8%提升至65.3%
大语言模型（LLMs）虽推动了自然语言处理发展，但存在幻觉问题，即生成看似合理却事实错误的内容。这一问题在视频-语言模型（VideoLLMs）中尤为突出，导致文本描述与视觉内容不符，产生多模态幻觉。本文针对结合ResNet视觉编码器与大语言模型的ResNetVLLM，提出两步协议：(1) 采用改进的Lynx模型检测生成描述与真实视频参考之间的语义一致性；(2) 利用检索增强生成（RAG）和动态构建的临时知识库进行幻觉缓解。所提出的ResNetVLLM-2模型通过外部知识交叉验证，提升了事实一致性。在ActivityNet-QA基准测试中，准确率从54.8%显著提升至65.3%，证明该检测与缓解策略能有效增强视频-语言模型的可靠性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have transformed natural language processing (NLP) tasks, but they suffer from hallucination, generating plausible yet factually incorrect content. This issue extends to Video-Language Models (VideoLLMs), where textual descriptions may inaccurately represent visual content, resulting in multi-modal hallucinations. In this paper, we address hallucination in ResNetVLLM, a video-language model combining ResNet visual encoders with LLMs. We introduce a two-step protocol: (1) a faithfulness detection strategy that uses a modified Lynx model to assess semantic alignment between generated captions and ground-truth video references, and (2) a hallucination mitigation strategy using Retrieval-Augmented Generation (RAG) with an ad-hoc knowledge base dynamically constructed during inference. Our enhanced model, ResNetVLLM-2, reduces multi-modal hallucinations by cross-verifying generated content against external knowledge, improving factual consistency. Evaluation on the ActivityNet-QA benchmark demonstrates a substantial accuracy increase from 54.8% to 65.3%, highlighting the effectiveness of our hallucination detection and mitigation strategies in enhancing video-language model reliability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。