用CLIP嵌入检测AI生成图像，少量数据即可达到高准确率
CLIP Embeddings for AI-Generated Image Detection: A Few-Shot Study with Lightweight Classifier
- 冻结CLIP提取视觉特征，仅微调轻量分类器
- 在CIFAKE上零样本达95%准确率，20%数据时仍保持85%
- 对广角照、油画等特定类型检测困难，适合关注生成内容安全的研究者
验证AI生成图像的真实性已成为社交媒体平台的严峻挑战。尽管视觉语言模型（如CLIP）在多模态表征上表现优异，但其在生成图像分类上的潜力因预训练阶段缺乏此类标签而未被充分探索。本文研究了CLIP嵌入是否蕴含生成痕迹。提出一种流程：使用冻结的CLIP模型提取视觉嵌入，输入轻量网络，仅微调最终分类器。在公开的CIFAKE基准测试中，无需语言推理即可实现95%准确率；仅使用20%的定制数据进行少样本适配，性能仍达85%。封闭源代码基线（Gemini-2.0）零样本表现最佳，但在特定风格上失效。值得注意的是，广角照片和油画等特定图像类型给分类带来显著挑战。结果揭示了该领域此前未被充分认识的困难，提出了更具体的新问题，值得进一步研究。
原文摘要 · Abstract (English)
Verifying the authenticity of AI-generated images presents a growing challenge on social media platforms these days. While vision-language models (VLMs) like CLIP outdo in multimodal representation, their capacity for AI-generated image classification is underexplored due to the absence of such labels during the pre-training process. This work investigates whether CLIP embeddings inherently contain information indicative of AI generation. A proposed pipeline extracts visual embeddings using a frozen CLIP model, feeds its embeddings to lightweight networks, and fine-tunes only the final classifier. Experiments on the public CIFAKE benchmark show the performance reaches 95% accuracy without language reasoning. Few-shot adaptation to curated custom with 20% of the data results in performance to 85%. A closed-source baseline (Gemini-2.0) has the best zero-shot accuracy yet fails on specific styles. Notably, some specific image types, such as wide-angle photographs and oil paintings, pose significant challenges to classification. These results indicate previously unexplored difficulties in classifying certain types of AI-generated images, revealing new and more specific questions in this domain that are worth further investigation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。