用少量样本快速识别新生成模型，解决伪造图像溯源难题。
Few-Shot Class-Incremental Model Attribution Using Learnable Representation From CLIP-ViT Features
- 基于CLIP-ViT特征设计可学习表示，融合多层视觉信息
- 在少样本条件下实现对新型生成模型的准确识别
- 适合需要快速响应新生成模型的安全检测场景
近年来，利用生成模型制造虚假图像已引发社会关注。为应对生成式人工智能模型的持续演进，仅检测合成图像已不够，模型溯源（MA）变得至关重要。然而，现有深度学习方法在面对新模型时需从头训练，耗时且依赖大量数据。本文提出一种新策略，将少样本类增量学习（FSCIL）机制引入MA任务，以识别新兴生成模型。不同于传统侧重高层语义的分类方法，MA需分析图像的低级特征如颜色、纹理。为此，我们利用CLIP-ViT不同层级特征构建可学习表示，并提出自适应融合模块（AIM），通过加权求和各块特征，提升模型辨识能力。大量实验表明，该方法能有效从已有模型扩展至近期出现的新模型。
原文摘要 · Abstract (English)
Recently, images that distort or fabricate facts using generative models have become a social concern. To cope with continuous evolution of generative artificial intelligence (AI) models, model attribution (MA) is necessary beyond just detection of synthetic images. However, current deep learning-based MA methods must be trained from scratch with new data to recognize unseen models, which is time-consuming and data-intensive. This work proposes a new strategy to deal with persistently emerging generative models. We adapt few-shot class-incremental learning (FSCIL) mechanisms for MA problem to uncover novel generative AI models. Unlike existing FSCIL approaches that focus on object classification using high-level information, MA requires analyzing low-level details like color and texture in synthetic images. Thus, we utilize a learnable representation from different levels of CLIP-ViT features. To learn an effective representation, we propose Adaptive Integration Module (AIM) to calculate a weighted sum of CLIP-ViT block features for each image, enhancing the ability to identify generative models. Extensive experiments show our method effectively extends from prior generative models to recent ones.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。