用图神经网络分析图文一致性，提升假信息检测准确率。
Evidence-Grounded Multimodal Misinformation Detection with Attention-Based GNNs
- 构建证据图与主张图，通过图神经网络比对图文匹配度
- 在测试集上达93.05%准确率，优于最佳LLM方法2.82%
- 适合需要高精度、小模型的可信信息验证场景
多模态脱离上下文（OOC）假信息指将真实图像搭配无关或误导性文字。这类信息检测困难，因需先理解语境再判断真伪。当前许多方法（如LLMs和LVLMs）缺乏上下文解析步骤，导致在无上下文时产生幻觉。本文提出一种基于图的方法，通过构建两个图表示：从在线文本证据生成的证据图，以及从标题中提取的主张图。利用图神经网络（GNNs）编码并比较这两个图，评估图文对的真实性。我们构建了专用数据集，对基线模型与主流LLMs在假信息检测任务上进行对比。实验表明，本方法在测试集上达到93.05%的检测准确率，优于第二佳方法（一个LLM）2.82%，支持更小、更专注的模型在该任务中的有效性。
原文摘要 · Abstract (English)
Multimodal out-of-context (OOC) misinformation is misinformation that repurposes real images with unrelated or misleading captions. Detecting such misinformation is challenging because it requires resolving the context of the claim before checking for misinformation. Many current methods, including LLMs and LVLMs, do not perform this contextualization step. LLMs hallucinate in absence of context or parametric knowledge. In this work, we propose a graph-based method that evaluates the consistency between the image and the caption by constructing two graph representations: an evidence graph, derived from online textual evidence, and a claim graph, from the claim in the caption. Using graph neural networks (GNNs) to encode and compare these representations, our framework then evaluates the truthfulness of image-caption pairs. We create datasets for our graph-based method, evaluate and compare our baseline model against popular LLMs on the misinformation detection task. Our method scores $93.05\%$ detection accuracy on the evaluation set and outperforms the second-best performing method (an LLM) by $2.82\%$, making a case for smaller and task-specific methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。