利用频域特征和增量学习,提升跨模型深度伪造检测准确率。
SPARK-IL: Spectral Retrieval-Augmented RAG for Knowledge-driven Deepfake Detection via Incremental Learning
- 双路径频域分析结合增量学习,增强泛化能力。
- 在19个生成模型上平均准确率达94.6%。
- 适合需要持续更新的实时深度伪造检测场景。
深度伪造图像检测面临重大挑战:针对特定生成器训练的检测器往往无法泛化到未见模型。尽管像素级伪影因模型而异,但频域特征更具一致性,为跨模型检测提供了可靠基础。为此,我们提出SPARK-IL,一种融合双路径频域分析与增量学习的检索增强框架。该框架采用部分冻结的ViT-L/14编码器提取语义表示,同时并行处理原始RGB像素嵌入。两条路径均通过多带傅里叶分解划分为四个频段,由基于专家混合的Kolmogorov-Arnold网络(KAN)进行频段专属变换,随后通过交叉注意力与残差连接融合频谱嵌入。推理时,融合嵌入从Milvus数据库中检索最相似的k个已标注签名,基于余弦相似度进行多数投票预测;同时,通过增量学习策略扩展数据库,并使用弹性权重巩固(EWC)保留先前学到的转换特征。在包含GAN、人脸换脸及扩散模型在内的19种生成模型上评估的UniversalFakeDetect基准测试中,SPARK-IL实现94.6%的平均准确率,代码将公开于https://github.com/HessenUPHF/SPARK-IL。
原文摘要 · Abstract (English)
Detecting AI-generated images remains a significant challenge because detectors trained on specific generators often fail to generalize to unseen models; however, while pixel-level artifacts vary across models, frequency-domain signatures exhibit greater consistency, providing a promising foundation for cross-generator detection. To address this, we propose SPARK-IL, a retrieval-augmented framework that combines dual-path spectral analysis with incremental learning by utilizing a partially frozen ViT-L/14 encoder for semantic representations alongside a parallel path for raw RGB pixel embeddings. Both paths undergo multi-band Fourier decomposition into four frequency bands, which are individually processed by Kolmogorov-Arnold Networks (KAN) with mixture-of-experts for band-specific transformations before the resulting spectral embeddings are fused via cross-attention with residual connections. During inference, this fused embedding retrieves the $k$ nearest labeled signatures from a Milvus database using cosine similarity to facilitate predictions via majority voting, while an incremental learning strategy expands the database and employs elastic weight consolidation to preserve previously learned transformations. Evaluated on the UniversalFakeDetect benchmark across 19 generative models -- including GANs, face-swapping, and diffusion methods -- SPARK-IL achieves a 94.6\% mean accuracy, with the code to be publicly released at https://github.com/HessenUPHF/SPARK-IL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。