利用视觉语言模型共识,实现少量标注数据下的假新闻检测。
CoVLM: Leveraging Consensus from Vision-Language Models for Semi-supervised Multi-modal Fake News Detection
- 基于标注数据自动设定阈值,生成未标注图文对的伪标签。
- 在真实场景下,仅用少量标注数据即达到领先性能。
- 适合数据稀缺、标签成本高的多模态假新闻检测任务。
本文针对现实世界中“图文错配”型虚假新闻检测难题,提出一种半监督框架CoVLM。现有方法依赖大量标注数据,但在实际中难以获取。相比之下,收集大规模未标注图文对更为可行。由于虚假新闻远少于真实新闻,数据集通常高度不平衡,加剧了检测难度。为此,CoVLM通过分析标注数据自适应设定阈值,为未标注样本生成鲁棒伪标签,实现自信伪标签的选择。在多个基准数据集上,该方法在挑战性条件下均优于当前最优模型,验证了其有效性。
原文摘要 · Abstract (English)
In this work, we address the real-world, challenging task of out-of-context misinformation detection, where a real image is paired with an incorrect caption for creating fake news. Existing approaches for this task assume the availability of large amounts of labeled data, which is often impractical in real-world, since it requires extensive manual intervention and domain expertise. In contrast, since obtaining a large corpus of unlabeled image-text pairs is much easier, here, we propose a semi-supervised protocol, where the model has access to a limited number of labeled image-text pairs and a large corpus of unlabeled pairs. Additionally, the occurrence of fake news being much lesser compared to the real ones, the datasets tend to be highly imbalanced, thus making the task even more challenging. Towards this goal, we propose a novel framework, Consensus from Vision-Language Models (CoVLM), which generates robust pseudo-labels for unlabeled pairs using thresholds derived from the labeled data. This approach can automatically determine the right threshold parameters of the model for selecting the confident pseudo-labels. Experimental results on benchmark datasets across challenging conditions and comparisons with state-of-the-art approaches demonstrate the effectiveness of our framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。