用新闻文章库替代图片搜索，定位新闻图片的拍摄时间和地点。
NewsRECON: News Article Retrieval for Image Contextualization
- 构建新闻文章检索系统，通过文章元数据推断图片时空信息。
- 在TARA数据集上超越已有方法，结合多模态大模型达新最优。
- 适用于跨地域时间的复杂场景，适合调查记者与事实核查者使用。
确定新闻图片的拍摄时间和地点对记者和法证专家核实信息、揭露虚假信息至关重要。尽管现有方法依赖反向图像搜索（RIS）引擎，但这些工具常无法返回结果，实用性受限。本文研究当RIS失效时的替代方案，探索利用新闻文章语料库作为补充：将图片与相关文章关联，从中提取日期和位置信息。我们评估了名为NewsRECON的新闻文章检索流程，其基于超过8.5万篇新闻文章的语料库。在TARA数据集上的实验表明，NewsRECON优于先前工作，并可与多模态大语言模型（MLLM）结合，在缺乏RIS证据的情况下实现新的最先进性能。此外，NewsRECON在5Pils-OOC基准上也表现出良好泛化能力，即使面对地理和时间上的偏差。代码已公开。
原文摘要 · Abstract (English)
Identifying when and where a news image was taken is crucial for journalists and forensic experts to produce credible stories and debunk misinformation. While many existing methods rely on reverse image search (RIS) engines, these tools often fail to return results, thereby limiting their practical applicability. In this work, we address the challenging scenario where RIS evidence is unavailable. We investigate the potential of news article corpora as an alternative to RIS, linking images to relevant articles to infer their dates and locations from article metadata. We evaluate the performance of a news article retrieval pipeline, NewsRECON, which leverages a corpus of over 85,000 articles. Experiments on the TARA dataset show that NewsRECON outperforms prior work and can be combined with a multimodal large language model (MLLM) to achieve new SOTA results in the absence of RIS evidence. Furthermore, NewsRECON generalizes to the 5Pils-OOC benchmark despite geographic and temporal shifts. We make our code available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。