通过挖掘多模态不一致特征,提升短视频平台假视频检测准确率
Consistency-aware Fake Videos Detection on Short Video Platforms
- 利用多模态不一致性作为判别线索,设计跨模态一致性学习模块
- 在FakeSV和FakeTT数据集上达到领先性能,显著优于现有方法
- 适合关注视频真实性验证与多模态分析的研究者与从业者
本文聚焦于短视频平台上的假新闻检测问题。尽管近年来相关研究取得显著进展,但检测准确率仍不理想,主要由于内容篡改与生成技术快速演进。现有方法通常直接融合原始视频数据与元数据后进行分类,但本研究发现:被篡改内容常表现出跨模态不一致性,而这一关键特征未被充分利用。为此,提出一种新范式,显式识别并利用跨模态矛盾作为判别线索。模型包含两个核心模块:跨模态一致性学习(CMCL)与多模态协同诊断(MMCD)。CMCL包括伪标签生成(PLG)与跨模态一致性诊断(CMCD),其中使用多模态大语言模型生成伪标签以评估语义一致性,并通过[CLS] token与余弦损失量化不一致性。MMCD通过多模态特征融合(MFF)与概率分数融合(PSF)整合信息,前者采用共注意力机制增强跨模态交互,后者利用Transformer实现综合特征融合。在主流基准数据集FakeSV与FakeTT上的大量实验表明,该模型在假视频检测任务中表现优异。
原文摘要 · Abstract (English)
This paper focuses to detect the fake news on the short video platforms. While significant research efforts have been devoted to this task with notable progress in recent years, current detection accuracy remains suboptimal due to the rapid evolution of content manipulation and generation technologies. Existing approaches typically employ a cross-modal fusion strategy that directly combines raw video data with metadata inputs before applying a classification layer. However, our empirical observations reveal a critical oversight: manipulated content frequently exhibits inter-modal inconsistencies that could serve as valuable discriminative features, yet remain underutilized in contemporary detection frameworks. Motivated by this insight, we propose a novel detection paradigm that explicitly identifies and leverages cross-modal contradictions as discriminative cues. Our approach consists of two core modules: Cross-modal Consistency Learning (CMCL) and Multi-modal Collaborative Diagnosis (MMCD). CMCL includes Pseudo-label Generation (PLG) and Cross-modal Consistency Diagnosis (CMCD). In PLG, a Multimodal Large Language Model is used to generate pseudo-labels for evaluating cross-modal semantic consistency. Then, CMCD extracts [CLS] tokens and computes cosine loss to quantify cross-modal inconsistencies. MMCD further integrates multimodal features through Multimodal Feature Fusion (MFF) and Probability Scores Fusion (PSF). MFF employs a co-attention mechanism to enhance semantic interactions across different modalities, while a Transformer is utilized for comprehensive feature fusion. Meanwhile, PSF further integrates the fake news probability scores obtained in the previous step. Extensive experiments on established benchmarks (FakeSV and FakeTT) demonstrate our model exhibits outstanding performance in Fake videos detection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。