用视觉语言模型生成逼真假标题数据，提升多模态虚假信息检测效果。
Latent Reconstruction from Generated Data for Multimodal Misinformation Detection
- 通过对抗性提示生成高保真假标题图像对，解决标注数据不足问题。
- 提出基于重建的LAMAR网络，在多个基准上达到新最佳性能。
- 适合关注虚假信息检测、生成数据与模型鲁棒性的研究者。
多模态虚假信息（如误标图像）在数字时代日益严重。由于缺乏大规模标注数据集，现有方法依赖于上下文错配或实体篡改生成的合成数据，但往往过于简单且不真实。为此，我们提出「MisCaption This!」框架，通过对抗性提示视觉语言模型生成高保真合成误标数据集。同时引入基于Transformer的「潜在多模态重建」（LAMAR），训练其重建真实标题的嵌入表示，作为强辅助信号指导检测。我们探索了端到端与大规模预训练等不同训练策略，以及直接、掩码、门控和注意力等多种融合机制。大量实验表明，基于「MisCaption This!」数据训练的模型在真实世界虚假信息上泛化能力更强；LAMAR在NewsCLIPpings、VERITE及新提出的VERITE 24/25基准上均取得新最优结果，验证了VLM生成数据与重建网络的有效性。代码已开源。
原文摘要 · Abstract (English)
Multimodal misinformation, such as miscaptioned images, where captions misrepresent an image's origin, context, or meaning, poses a growing challenge in the digital age. Due to the scarcity of large-scale annotated datasets for multimodal misinformation detection (MMD), recent approaches rely on synthetic training data created via out-of-context pairings or named entity manipulations (e.g., altering names, dates, or locations). However, these often yield simplistic, unrealistic examples, which limits their utility as training examples. To address this, we introduce "MisCaption This!", a framework for generating high-fidelity synthetic miscaptioned datasets through Adversarial Prompting of Vision-Language Models (VLMs). Additionally, we introduce "Latent Multimodal Reconstruction" (LAMAR), a Transformer-based network trained to reconstruct the embeddings of truthful captions, providing a strong auxiliary signal to guide detection. We explore various training strategies (end-to-end vs. large-scale pre-training) and integration mechanisms (direct, mask, gate, and attention). Extensive experiments show that models trained on "MisCaption This!" data generalize better to real-world misinformation, while LAMAR achieves new state-of-the-art on NewsCLIPpings, VERITE, and the newly introduced VERITE 24/25 benchmark; highlighting the efficacy of VLM-generated data and reconstruction-based networks for advancing MMD. Our code is available at https://github.com/stevejpapad/miscaptioned-image-reconstruction
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。