用多粒度检索和多智能体推理,精准识别图文错配的虚假信息。
EXCLAIM: An Explainable Cross-Modal Agentic System for Misinformation Detection with Hierarchical Retrieval
- 构建多粒度多模态索引,从事件到实体分层检索知识
- 在真实数据集上比顶尖方法高4.3%准确率
- 可生成解释性结论,适合舆情监测与内容审核
虚假信息持续威胁当今的信息生态,深刻影响公众认知与行为。其中,上下文错位(OOC)虚假信息尤为隐蔽——通过将真实图片与误导性文字搭配扭曲原意。现有检测方法多依赖图像-文本对的粗粒度相似性,难以捕捉细微矛盾且缺乏可解释性。尽管多模态大模型在视觉推理与解释生成方面表现卓越,却尚未具备处理复杂、细粒度跨模态差异的能力。为此,我们提出EXCLAIM,一种基于检索的框架,通过多粒度的多模态事件与实体索引,引入外部知识。该方法结合多粒度上下文分析与多智能体推理架构,系统评估多模态新闻内容的一致性与完整性。大量实验验证了EXCLAIM的有效性与鲁棒性,在真实数据集上相较现有最优方法提升4.3%的准确率,同时提供可解释、可操作的洞察。
原文摘要 · Abstract (English)
Misinformation continues to pose a significant challenge in today's information ecosystem, profoundly shaping public perception and behavior. Among its various manifestations, Out-of-Context (OOC) misinformation is particularly obscure, as it distorts meaning by pairing authentic images with misleading textual narratives. Existing methods for detecting OOC misinformation predominantly rely on coarse-grained similarity metrics between image-text pairs, which often fail to capture subtle inconsistencies or provide meaningful explainability. While multi-modal large language models (MLLMs) demonstrate remarkable capabilities in visual reasoning and explanation generation, they have not yet demonstrated the capacity to address complex, fine-grained, and cross-modal distinctions necessary for robust OOC detection. To overcome these limitations, we introduce EXCLAIM, a retrieval-based framework designed to leverage external knowledge through multi-granularity index of multi-modal events and entities. Our approach integrates multi-granularity contextual analysis with a multi-agent reasoning architecture to systematically evaluate the consistency and integrity of multi-modal news content. Comprehensive experiments validate the effectiveness and resilience of EXCLAIM, demonstrating its ability to detect OOC misinformation with 4.3% higher accuracy compared to state-of-the-art approaches, while offering explainable and actionable insights.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。