用注意力机制强化对伪造痕迹的捕捉，提升ViT在深伪检测中的表现。
FakeFormer: Efficient Vulnerability-Driven Transformers for Generalisable Deepfake Detection
- 引入针对伪造痕迹的显式注意力学习，增强ViT对局部异常的敏感性。
- 在6个数据集上超越现有方法，且无需大规模训练数据。
- 适合关注高效、泛化强的深伪检测研究者与应用开发者。
近期，视觉变换器（ViTs）在图像分类任务中表现出色，但在深伪检测领域仍逊于卷积神经网络（CNNs）。本文分析发现，与CNN相比，ViT难以建模深伪图像中常见的局部伪造痕迹。为此，提出FakeFormer框架，通过面向伪造脆弱区域的显式注意力机制，增强ViT对细微不一致信息的提取能力。在多个知名数据集（包括FF++、Celeb-DF、WildDeepfake、DFD、DFDCP和DFDC）上的实验表明，FakeFormer在泛化性能和计算开销方面均优于当前最优方法，且无需依赖大规模训练数据。代码已开源。
原文摘要 · Abstract (English)
Recently, Vision Transformers (ViTs) have achieved unprecedented effectiveness in the general domain of image classification. Nonetheless, these models remain underexplored in the field of deepfake detection, given their lower performance as compared to Convolution Neural Networks (CNNs) in that specific context. In this paper, we start by investigating why plain ViT architectures exhibit a suboptimal performance when dealing with the detection of facial forgeries. Our analysis reveals that, as compared to CNNs, ViT struggles to model localized forgery artifacts that typically characterize deepfakes. Based on this observation, we propose a deepfake detection framework called FakeFormer, which extends ViTs to enforce the extraction of subtle inconsistency-prone information. For that purpose, an explicit attention learning guided by artifact-vulnerable patches and tailored to ViTs is introduced. Extensive experiments are conducted on diverse well-known datasets, including FF++, Celeb-DF, WildDeepfake, DFD, DFDCP, and DFDC. The results show that FakeFormer outperforms the state-of-the-art in terms of generalization and computational cost, without the need for large-scale training datasets. The code is available at \url{https://github.com/10Ring/FakeFormer}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。