用时空注意力捕捉嘴部不自然动作,识别语音同步类深度伪造视频。
Detecting Lip-Syncing Deepfakes: Vision Temporal Transformer for Analyzing Mouth Inconsistencies
- 结合视觉时序变换器与多头交叉注意力,分析嘴部区域的时空不一致。
- 在自建数据集上达到当前最佳检测准确率,对五种主流生成模型有效。
- 适合关注多媒体安全、深度伪造检测的研究者与应用开发者。
深度伪造是通过人工智能技术篡改原始内容生成的逼真但虚假的图像、视频或音频。其中,语音同步类深度伪造最为棘手:利用AI模型将人物口型合成以匹配修改或全新的音频,其伪造痕迹局限于嘴部区域,更加隐蔽且难以察觉。本文提出LIPINC-V2检测框架,采用视觉时序变换器与多头交叉注意力机制,识别嘴部区域在相邻帧间持续存在的时空不一致性。该模型能有效捕捉短时与长时口型运动异常。我们构建了新数据集LipSyncTIMIT,使用五种前沿语音同步模型生成,模拟真实场景。在所提数据集及两个基准数据集上的大量实验表明,本方法性能领先现有技术。代码与数据集已公开于https://github.com/skrantidatta/LIPINC-V2。
原文摘要 · Abstract (English)
Deepfakes are AI-generated media in which the original content is digitally altered to create convincing but manipulated images, videos, or audio. Among the various types of deepfakes, lip-syncing deepfakes are one of the most challenging deepfakes to detect. In these videos, a person's lip movements are synthesized to match altered or entirely new audio using AI models. Therefore, unlike other types of deepfakes, the artifacts in lip-syncing deepfakes are confined to the mouth region, making them more subtle and, thus harder to discern. In this paper, we propose LIPINC-V2, a novel detection framework that leverages a combination of vision temporal transformer with multihead cross-attention to detect lip-syncing deepfakes by identifying spatiotemporal inconsistencies in the mouth region. These inconsistencies appear across adjacent frames and persist throughout the video. Our model can successfully capture both short-term and long-term variations in mouth movement, enhancing its ability to detect these inconsistencies. Additionally, we created a new lip-syncing deepfake dataset, LipSyncTIMIT, which was generated using five state-of-the-art lip-syncing models to simulate real-world scenarios. Extensive experiments on our proposed LipSyncTIMIT dataset and two other benchmark deepfake datasets demonstrate that our model achieves state-of-the-art performance. The code and the dataset are available at https://github.com/skrantidatta/LIPINC-V2 .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。