用大模型统一检测与解释AI造假图像,跨模型效果更优。
Toward Generalizable Forgery Detection and Reasoning
- 用多模态大模型结合视觉特征与推理,统一处理检测与解释。
- 在120K张跨模型伪造图像上,检测准确率显著领先现有方法。
- 适合安全、内容审核等需要可解释性检测的场景。
准确且可解释的AI生成图像检测对缓解人工智能滥用风险至关重要。然而,生成模型间存在显著领域差异,导致通用化检测模型难建。此外,由于AI生成图像每个像素均被合成,传统基于显著性的伪造解释方法不适用。为此,我们提出统一的伪造检测与推理任务(FDR-Task),利用多模态大语言模型(MLLMs)通过伪造属性推理实现精准检测。为支持该任务,我们构建了大规模多模态伪造推理数据集(MMFR-Dataset),包含120K张来自10种生成模型的图像,以及378K条伪造属性推理标注,支持全面评估FDR-Task。同时提出FakeReasoning框架,包含三个关键组件:1)融合CLIP与DINO的双分支视觉编码器,捕捉高层语义与低层伪影;2)伪造感知特征融合模块,利用DINO注意力图与交叉注意力机制引导MLLM关注伪造线索;3)分类概率映射器,结合语言建模与检测任务,提升整体性能。在多个生成模型上的实验表明,FakeReasoning不仅具备强泛化能力,且在检测与推理任务上均优于当前最优方法。代码已开源:https://github.com/PRIS-CV/FakeReasoning。
原文摘要 · Abstract (English)
Accurate and interpretable detection of AI-generated images is essential for mitigating risks associated with AI misuse. However, the substantial domain gap among generative models makes it challenging to develop a generalizable forgery detection model. Moreover, since every pixel in an AI-generated image is synthesized, traditional saliency-based forgery explanation methods are not well suited for this task. To address these challenges, we formulate detection and explanation as a unified Forgery Detection and Reasoning task (FDR-Task), leveraging Multi-Modal Large Language Models (MLLMs) to provide accurate detection through reliable reasoning over forgery attributes. To facilitate this task, we introduce the Multi-Modal Forgery Reasoning dataset (MMFR-Dataset), a large-scale dataset containing 120K images across 10 generative models, with 378K reasoning annotations on forgery attributes, enabling comprehensive evaluation of the FDR-Task. Furthermore, we propose FakeReasoning, a forgery detection and reasoning framework with three key components: 1) a dual-branch visual encoder that integrates CLIP and DINO to capture both high-level semantics and low-level artifacts; 2) a Forgery-Aware Feature Fusion Module that leverages DINO's attention maps and cross-attention mechanisms to guide MLLMs toward forgery-related clues; 3) a Classification Probability Mapper that couples language modeling and forgery detection, enhancing overall performance. Experiments across multiple generative models demonstrate that FakeReasoning not only achieves robust generalization but also outperforms state-of-the-art methods on both detection and reasoning tasks. The code is available at: https://github.com/PRIS-CV/FakeReasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。