用多模态特征金字塔+Transformer精准定位深度伪造片段
Multi-modal Deepfake Detection and Localization with FPN-Transformer
- 融合音频视频自监督特征，通过局部注意力建模跨模态时序依赖
- 在IJCAI'25基准上实现0.7535的检测定位得分，支持帧级精确定位
- 适合需要高精度伪造内容识别的数字安全与媒体审核场景
生成对抗网络（GAN）和扩散模型的快速发展催生了高度逼真的深度伪造内容，严重威胁音视频领域的数字信任。尽管单模态检测方法取得进展，但难以利用跨模态关联并精确定位伪造片段，限制了其在复杂精细篡改下的实用性。为此，我们提出基于特征金字塔-Transformer（FPN-Transformer）的多模态深度伪造检测与定位框架，解决了跨模态泛化与时间边界回归的关键问题。该方法采用预训练自监督模型（WavLM用于音频，CLIP用于视频）提取分层时序特征，通过R-TLM块构建多尺度特征金字塔，结合局部注意力机制实现跨上下文时序依赖联合分析。双分支预测头同时输出伪造概率并优化篡改段的时间偏移，实现帧级定位精度。我们在IJCAI'25 DDL-AV基准测试集上评估，跨模态检测与定位最终得分为0.7535，在复杂环境下表现优异。实验验证了方法有效性，为通用深度伪造检测提供了新思路。代码已开源。
原文摘要 · Abstract (English)
The rapid advancement of generative adversarial networks (GANs) and diffusion models has enabled the creation of highly realistic deepfake content, posing significant threats to digital trust across audio-visual domains. While unimodal detection methods have shown progress in identifying synthetic media, their inability to leverage cross-modal correlations and precisely localize forged segments limits their practicality against sophisticated, fine-grained manipulations. To address this, we introduce a multi-modal deepfake detection and localization framework based on a Feature Pyramid-Transformer (FPN-Transformer), addressing critical gaps in cross-modal generalization and temporal boundary regression. The proposed approach utilizes pre-trained self-supervised models (WavLM for audio, CLIP for video) to extract hierarchical temporal features. A multi-scale feature pyramid is constructed through R-TLM blocks with localized attention mechanisms, enabling joint analysis of cross-context temporal dependencies. The dual-branch prediction head simultaneously predicts forgery probabilities and refines temporal offsets of manipulated segments, achieving frame-level localization precision. We evaluate our approach on the test set of the IJCAI'25 DDL-AV benchmark, showing a good performance with a final score of 0.7535 for cross-modal deepfake detection and localization in challenging environments. Experimental results confirm the effectiveness of our approach and provide a novel way for generalized deepfake detection. Our code is available at https://github.com/Zig-HS/MM-DDL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。