arXiv:2410.16592cs.LGcs.CL2024-10

首个可同时分析视频三模态的假信息检测系统,有效提升短视頻真偽判斷能力。

ViMGuard: A Novel Multi-Modal System for Video Misinformation Guarding

  • 通过视觉与音频掩码自编码器分析视频意图,判断是否含信息性陈述。
  • 采用检索增强生成技术验证口语内容的真实性,准确率超越现有方法。
  • 开源工具已集成至浏览器插件,便于实际应用与后续研究。

社交媒体和短视频(SFV)的兴起为虚假信息传播提供了温床。尽管大语言模型在文本假信息检测方面已有进展,但对短视频中多模态信息的自动识别仍处于未充分研究状态。由于短视频包含文字、图像和非语言音频三种模态,其分析远比纯文本复杂。本文提出首个基于深度学习的三模态假信息检测框架——ViMGuard,可同时分析视频的视觉、音频与语言内容。该系统由双组件构成:首先,视频与音频掩码自编码器用于解析视觉与非语言音频元素,判断视频是否具有信息性意图;若判定为信息性内容,则进入第二阶段——检索增强生成系统,对语音内容进行事实准确性验证。实验表明,ViMGuard在多个基准测试中优于三种先进检测器,树立了短视频事实核查的新标准。为促进进一步研究,系统已部署为Chrome扩展,并开源于GitHub。

原文摘要 · Abstract (English)

The rise of social media and short-form video (SFV) has facilitated a breeding ground for misinformation. With the emergence of large language models, significant research has gone into curbing this misinformation problem with automatic false claim detection for text. Unfortunately, the automatic detection of misinformation in SFV is a more complex problem that remains largely unstudied. While text samples are monomodal (only containing words), SFVs comprise three different modalities: words, visuals, and non-linguistic audio. In this work, we introduce Video Masked Autoencoders for Misinformation Guarding (ViMGuard), the first deep-learning architecture capable of fact-checking an SFV through analysis of all three of its constituent modalities. ViMGuard leverages a dual-component system. First, Video and Audio Masked Autoencoders analyze the visual and non-linguistic audio elements of a video to discern its intention; specifically whether it intends to make an informative claim. If it is deemed that the SFV has informative intent, it is passed through our second component: a Retrieval Augmented Generation system that validates the factual accuracy of spoken words. In evaluation, ViMGuard outperformed three cutting-edge fact-checkers, thus setting a new standard for SFV fact-checking and marking a significant stride toward trustworthy news on social platforms. To promote further testing and iteration, VimGuard was deployed into a Chrome extension and all code was open-sourced on GitHub.

视频生成多模态假信息检测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。