用视觉语言模型提取多模态特征，提升假信息检测准确率
Multimodal Fact-Checking with Vision Language Models: A Probing Classifier based Solution with Embedding Strategies
- 从VLM中提取图文嵌入，输入神经探测分类器进行真假判断
- 图文分开编码再融合，比直接用VLM嵌入效果更好
- 新分类器显著优于KNN和SVM，适合多模态内容验证
本研究评估视觉语言模型（VLMs）在多模态事实核查中的表示与利用能力。具体而言，探究引入多模态内容是否优于纯文本模型，以及VLM如何利用文本与图像信息提升虚假信息检测效果。为此，提出一种基于探测分类器的解决方案：从选定VLM的最后隐藏层提取嵌入，并输入神经探测分类器进行多类别真伪分类。在两个事实核查数据集上的实验表明，尽管多模态可提升性能，但分别提取文本与图像编码器的嵌入并融合，效果优于直接使用VLM嵌入。此外，所提神经分类器在利用提取嵌入方面显著优于KNN与SVM基线，凸显其在多模态事实核查中的有效性。
原文摘要 · Abstract (English)
This study evaluates the effectiveness of Vision Language Models (VLMs) in representing and utilizing multimodal content for fact-checking. To be more specific, we investigate whether incorporating multimodal content improves performance compared to text-only models and how well VLMs utilize text and image information to enhance misinformation detection. Furthermore we propose a probing classifier based solution using VLMs. Our approach extracts embeddings from the last hidden layer of selected VLMs and inputs them into a neural probing classifier for multi-class veracity classification. Through a series of experiments on two fact-checking datasets, we demonstrate that while multimodality can enhance performance, fusing separate embeddings from text and image encoders yielded superior results compared to using VLM embeddings. Furthermore, the proposed neural classifier significantly outperformed KNN and SVM baselines in leveraging extracted embeddings, highlighting its effectiveness for multimodal fact-checking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。