融合知识图谱与多模态信息,提升假新闻检测准确性
KGAlign: Joint Semantic-Structural Knowledge Encoding for Multimodal Fake News Detection
- 通过局部物体注意力+全局图像语义捕捉视觉细节
- 在MIMIC-2023数据集上达到89.6%准确率,优于现有方法
- 适合研究多模态推理与虚假信息检测的学者
假新闻检测因文本误导、图像篡改与外部知识推理的复杂交互而面临挑战。现有方法常仅关注全局图像上下文,忽略局部物体细节,且未能有效融入外部知识与实体关系以实现深层语义理解。为此,我们提出一种新型多模态假新闻检测框架,整合视觉、文本与知识表示。利用自底向上注意力捕获细粒度物体特征,CLIP提取全局图像语义,RoBERTa进行上下文感知文本编码,并从知识图谱中检索并自适应选择相关实体以增强知识利用。融合后的多模态特征通过Transformer分类器预测新闻真伪。实验表明,该模型在MIMIC-2023数据集上达到89.6%准确率,显著优于近期方法。所提邻域选择机制与多模态融合策略验证了知识引导的语义验证范式有效性。
原文摘要 · Abstract (English)
Fake news detection remains a challenging problem due to the complex interplay between textual misinformation, manipulated images, and external knowledge reasoning. While existing approaches have achieved notable results in verifying veracity and cross-modal consistency, two key challenges persist: (1) Existing methods often consider only the global image context while neglecting local object-level details, and (2) they fail to incorporate external knowledge and entity relationships for deeper semantic understanding. To address these challenges, we propose a novel multi-modal fake news detection framework that integrates visual, textual, and knowledge-based representations. Our approach leverages bottom-up attention to capture fine-grained object details, CLIP for global image semantics, and RoBERTa for context-aware text encoding. We further enhance knowledge utilization by retrieving and adaptively selecting relevant entities from a knowledge graph. The fused multi-modal features are processed through a Transformer-based classifier to predict news veracity. Experimental results demonstrate that our model outperforms recent approaches, showcasing the effectiveness of neighbor selection mechanism and multi-modal fusion for fake news detection. Our proposal introduces a new paradigm: knowledge-grounded multimodal reasoning. By integrating explicit entity-level selection and NLI-guided filtering, we shift fake news detection from feature fusion to semantically grounded verification. For reproducibility and further research, the source code is publicly at \href{https://github.com/latuanvinh1998/KGAlign}{github.com/latuanvinh1998/KGAlign}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。