针对中英双语多媒体假信息,实现精准定位与可解释检测。
Towards Explainable Bilingual Multimodal Misinformation Detection and Localization
- 联合检测图像区域篡改与跨模态跨语言不一致
- 在10.4万样本上实现分类准确率提升8.9%、定位准确率提升15.9%
- 首次用GRPO优化生成解释,适合多语言内容审核场景
随着多媒体内容日益逼真,假信息愈发隐蔽,尤其在图文配中英双语字幕的新闻中。此类内容常包含局部图像编辑与跨语言不一致,共同扭曲语义却保持表面可信。我们提出BiMi框架,实现区域级定位、跨模态与跨语言一致性检测及自然语言解释。为增强泛化能力,引入在线检索模块补充实时外部上下文。我们进一步发布BiMiBench,一个大规模综合基准,包含104,000个真实篡改样本,覆盖视觉与语言模态。为提升可解释性,首次在该领域应用组相对策略优化(GRPO)改进解释质量。大量实验表明,BiMi在分类准确率上比强基线最高提升8.9%,定位准确率提升15.9%,解释BERTScore提升2.5,显著推进真实多语言假信息检测的性能。代码、模型与数据集将开源。
原文摘要 · Abstract (English)
The increasing realism of multimodal content has made misinformation more subtle and harder to detect, especially in news media where images are frequently paired with bilingual (e.g., Chinese-English) subtitles. Such content often includes localized image edits and cross-lingual inconsistencies that jointly distort meaning while remaining superficially plausible. We introduce BiMi, a bilingual multimodal framework that jointly performs region-level localization, cross-modal and cross-lingual consistency detection, and natural language explanation for misinformation analysis. To support generalization, BiMi integrates an online retrieval module that supplements model reasoning with up-to-date external context. We further release BiMiBench, a large-scale and comprehensive benchmark constructed by systematically editing real news images and subtitles, comprising 104,000 samples with realistic manipulations across visual and linguistic modalities. To enhance interpretability, we apply Group Relative Policy Optimization (GRPO) to improve explanation quality, marking the first use of GRPO in this domain. Extensive experiments demonstrate that BiMi outperforms strong baselines by up to +8.9 in classification accuracy, +15.9 in localization accuracy, and +2.5 in explanation BERTScore, advancing state-of-the-art performance in realistic, multilingual misinformation detection. Code, models, and datasets will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。