同时预测图像真实上下文与标题真假，提升谣言识别效果。
COVE: COntext and VEracity prediction for out-of-context images
- 先预测图像真实上下文，再据此判断标题真伪。
- 上下文预测准确率超现有模型5个百分点以上，标题真伪判断在真实数据上更优。
- 预测的上下文可复用且易懂，适合人工验证新标题。
脱离原上下文的图像已成为多模态虚假信息最常见的形式。揭穿这类信息需要（1）还原图像的真实上下文，（2）验证其标题的真实性。然而，现有自动化事实核查方法未能明确处理这两个目标。本文提出COVE，一种新方法：先预测图像的真实上下文，再利用该上下文判断标题真伪。COVE在所有上下文预测指标上均超越当前最佳模型，性能提升常超过5个百分点；在合成数据上与最优真伪预测模型相当，在真实数据上表现更优，证明了两任务顺序联合的益处。此外，我们开展的人类研究显示，所预测的上下文是可复用、可解释的验证工具，可用于检验同一图像的新标题。代码与数据已公开。
原文摘要 · Abstract (English)
Images taken out of their context are the most prevalent form of multimodal misinformation. Debunking them requires (1) providing the true context of the image and (2) checking the veracity of the image's caption. However, existing automated fact-checking methods fail to tackle both objectives explicitly. In this work, we introduce COVE, a new method that predicts first the true COntext of the image and then uses it to predict the VEracity of the caption. COVE beats the SOTA context prediction model on all context items, often by more than five percentage points. It is competitive with the best veracity prediction models on synthetic data and outperforms them on real-world data, showing that it is beneficial to combine the two tasks sequentially. Finally, we conduct a human study that reveals that the predicted context is a reusable and interpretable artifact to verify new out-of-context captions for the same image. Our code and data are made available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。