arXiv:2409.02664cs.CV2024-09AAAI被引 45

用视觉语言模型检测深度伪造,无需训练就能跨数据集有效识别。

Standing on the Shoulders of Giants: Reprogramming Visual-Language Model for General Deepfake Detection

论文配图:Standing on the Shoulders of Giants: Reprogramming Visual-Language Model for General Deepfake Detection
图 1 · 摘自论文原文
  • 通过可学习的视觉扰动和自适应文本提示,改造预训练模型。
  • 在跨数据集测试中AUC超88%,对新生成模型也表现稳定。
  • 参数少、部署快,适合实际场景快速部署。

深度伪造人脸的泛滥对日常生活带来巨大潜在风险。尽管近年来检测技术进步显著,现有方法在面对未见过的数据集或新兴生成模型产生的伪造内容时,泛化能力仍受限。受视觉语言模型(VLMs)零样本优势启发,本文提出一种新方法,将预训练的VLM(如CLIP)重新用于通用深度伪造检测。该方法基于模型重编程范式,仅通过输入扰动即可操控模型输出,无需调整内部参数。首先,使用可学习的视觉扰动优化特征提取;其次,利用人脸嵌入生成样本级自适应文本提示,提升检测性能。在多个主流基准数据集上的实验表明:(1)跨数据集与跨操作类型的检测性能显著且一致提升(如从FF++到WildDeepfake设置下AUC超过88%);(2)以极少的可训练参数实现优异表现,具备良好的实用前景。

原文摘要 · Abstract (English)

The proliferation of deepfake faces poses huge potential negative impacts on our daily lives. Despite substantial advancements in deepfake detection over these years, the generalizability of existing methods against forgeries from unseen datasets or created by emerging generative models remains constrained. In this paper, inspired by the zero-shot advantages of Vision-Language Models (VLMs), we propose a novel approach that repurposes a well-trained VLM for general deepfake detection. Motivated by the model reprogramming paradigm that manipulates the model prediction via input perturbations, our method can reprogram a pre-trained VLM model (e.g., CLIP) solely based on manipulating its input without tuning the inner parameters. First, learnable visual perturbations are used to refine feature extraction for deepfake detection. Then, we exploit information of face embedding to create sample-level adaptative text prompts, improving the performance. Extensive experiments on several popular benchmark datasets demonstrate that (1) the cross-dataset and cross-manipulation performances of deepfake detection can be significantly and consistently improved (e.g., over 88\% AUC in cross-dataset setting from FF++ to WildDeepfake); (2) the superior performances are achieved with fewer trainable parameters, making it a promising approach for real-world applications.

深度伪造视觉语言模型零样本检测模型重编程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。