解决多模态假信息识别中缺失模态问题,提升模型鲁棒性。
Towards Robust and Realible Multimodal Misinformation Recognition with Incomplete Modality

- 通过多专家协作动态补全缺失模态信息。
- 在跨语言数据集上显著优于现有方法,准确率提升明显。
- 适合真实场景中模态不完整的信息识别任务。
随着社交媒体上海量多模态虚假内容的出现,多模态假信息识别已成为迫切任务。以往研究主要关注复杂特征提取与融合,以从多模态内容中学习判别性信息。然而,在真实应用中,多媒体新闻在传播过程中可能自然丢失部分信息,导致模态不完整,严重影响现有模型的泛化与鲁棒性。为此,我们提出一种新颖、通用且鲁棒的多模态融合策略——多专家模态缺失学习网络(MMLNet),该方法简单而有效,包含三个关键步骤:(1) 多专家协同推理,通过多个专家动态利用互补信息补偿缺失模态;(2) 模态缺失适配器,基于新特征分布弥补缺失信息;(3) 模态缺失学习,采用标签感知自适应加权策略,结合对比学习构建鲁棒表示。我们在两个语言的三个真实世界基准上评估了MMLNet,结果表明其性能显著优于当前最优方法,同时保持相对简洁。MMLNet有效提升了因信息传播导致模态缺失场景下的假信息识别准确率,从而更有效地遏制恶意虚假信息的扩散。代码已公开于 https://github.com/zhyhome/MMLNet。
原文摘要 · Abstract (English)
Multimodal Misinformation Recognition has become an urgent task with the emergence of huge multimodal fake content on social media platforms. Previous studies mainly focus on complex feature extraction and fusion to learn discriminative information from multimodal content. However, in real-world applications, multimedia news may naturally lose some information during dissemination, resulting in modality incompleteness, which is detrimental to the generalization and robustness of existing models. To this end, we propose a novel generic and robust multimodal fusion strategy, termed Multi-expert Modality-incomplete Learning Network (MMLNet), which is simple yet effective. It consists of three key steps: (1) Multi-Expert Collaborative Reasoning to compensate for missing modalities by dynamically leveraging complementary information through multiple experts. (2) Incomplete Modality Adapters compensates for the missing information by leveraging the new feature distribution. (3) Modality Missing Learning leveraging an label-aware adaptive weighting strategy to learn a robust representation with contrastive learning. We evaluate MMLNet on three real-world benchmarks across two languages, demonstrating superior performance compared to state-of-the-art methods while maintaining relative simplicity. By ensuring the accuracy of misinformation recognition in incomplete modality scenarios caused by information propagation, MMLNet effectively curbs the spread of malicious misinformation. Code is publicly available at https://github.com/zhyhome/MMLNet.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。