arXiv:2411.19715cs.CVcs.CR2024-11TPAMI被引 2

用轻量适配器让CLIP精准识别伪造人脸,通用性强且只需570万参数。

Forensics Adapter: Unleashing CLIP for Generalizable Face Forgery Detection

  • 设计适配器捕捉伪造人脸特有的融合边界特征,引导任务专有学习。
  • 在六个标准数据集上表现优于现有方法,仅用570万参数即达顶尖水平。
  • 适配器结构保留CLIP通用性,适合做基于CLIP的伪造检测基准研究。

我们提出Forensics Adapter,一种适配器网络,可将CLIP转化为高效且泛化能力强的面部伪造检测器。尽管CLIP具备高度通用性,但将其用于面部伪造检测面临挑战:伪造相关知识与大量无关知识混杂。现有方法仅将CLIP作为特征提取器,缺乏任务特定适应,限制了性能。为此,我们引入适配器,学习伪造人脸特有的融合边界痕迹,并通过任务特定目标进行引导。同时,采用专门的交互策略增强CLIP视觉标记,实现CLIP与适配器间知识传递。由于适配器与CLIP并行,其通用性得以保留,自然保障了在面部伪造检测中的强泛化能力。本方法仅需570万可训练参数,便在六个标准数据集上取得卓越性能。此外,我们提出Forensics Adapter++,通过新提出的伪造感知提示学习策略引入文本模态,相比原版再提升1.3%性能。我们认为所提方法可作为未来基于CLIP的面部伪造检测研究的基准。代码已开源:https://github.com/OUC-VAS/ForensicsAdapter。

原文摘要 · Abstract (English)

We describe Forensics Adapter, an adapter network designed to transform CLIP into an effective and generalizable face forgery detector. Although CLIP is highly versatile, adapting it for face forgery detection is non-trivial as forgery-related knowledge is entangled with a wide range of unrelated knowledge. Existing methods treat CLIP merely as a feature extractor, lacking task-specific adaptation, which limits their effectiveness. To address this, we introduce an adapter to learn face forgery traces -- the blending boundaries unique to forged faces, guided by task-specific objectives. Then we enhance the CLIP visual tokens with a dedicated interaction strategy that communicates knowledge across CLIP and the adapter. Since the adapter is alongside CLIP, its versatility is highly retained, naturally ensuring strong generalizability in face forgery detection. {With only $\textbf{5.7M}$ trainable parameters, our method achieves superior performance across six standard datasets.} Additionally, we describe Forensics Adapter++, an extended method that incorporates textual modality via a newly proposed forgery-aware prompt learning strategy. This extension leads to a further $\textbf{1.3\%}$ performance boost over the original Forensics Adapter. We believe the proposed methods can serve as a baseline for future CLIP-based face forgery detection methods. The code has been released at https://github.com/OUC-VAS/ForensicsAdapter.

伪造检测CLIP适配器轻量模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。