用视觉语言模型验证新闻图文中的实体一致性,识别跨模态虚假信息。
Verifying Cross-modal Entity Consistency in News using Vision-language Models
- 基于大视觉语言模型设计提示策略,利用网络爬取参考图验证实体
- 在人物和事件验证上准确率提升,位置与事件验证优于基线
- 构建三个新闻实体验证数据集,适合反虚假信息研究者使用
网络已成为重要信息来源,但也被用于传播多模态虚假信息,如图文不一致。现有方法或仅评估图像与全文的一致性,忽略具体实体关系;或关注无关通用实体。目前极少工作专门验证新闻中人物、地点、事件的跨模态一致性。本文提出基于大视觉语言模型的框架LVLM4CEC,通过有效提示策略与网络爬取的参考图像,验证新闻图文中的实体一致性。我们扩展了三个现有数据集并提供人工标注真值。实验表明,该方法在人物和事件验证上表现更优,且在位置与事件验证上超越基线。代码与数据集已开源于GitHub。
原文摘要 · Abstract (English)
The web has become a crucial source of information, but it is also used to spread disinformation, often conveyed through multiple modalities like images and text. The identification of inconsistent cross-modal information, in particular entities such as persons, locations, and events, is critical to detect disinformation. Previous works either identify out-of-context disinformation by assessing the consistency of images to the whole document, neglecting relations of individual entities, or focus on generic entities that are not relevant to news. So far, only few approaches have addressed the task of validating entity consistency between images and text in news. However, the potential of large vision-language models (LVLMs) has not been explored yet. In this paper, we propose an LVLM-based framework for verifying Cross-modal Entity Consistency~(LVLM4CEC), to assess whether persons, locations and events in news articles are consistent across both modalities. We suggest effective prompting strategies for LVLMs for entity verification that leverage reference images crawled from web. Moreover, we extend three existing datasets for the task of entity verification in news providing manual ground-truth data. Our results show the potential of LVLMs for automating cross-modal entity verification, showing improved accuracy in identifying persons and events when using evidence images. Moreover, our method outperforms a baseline for location and event verification in documents. The datasets and source code are available on GitHub at https://github.com/TIBHannover/LVLM4CEC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。