多模态大模型可做深度伪造检测,部分表现超越传统方法。
Can Multi-modal (reasoning) LLMs work as deepfake detectors?
- 用提示调优和推理路径分析,测试12个多模态大模型的检测能力。
- 顶尖模型零样本下表现媲美传统方法,跨数据集泛化能力强。
- 新版本和推理能力未必提升,模型规模在某些情况下有帮助。
深度伪造检测在生成模型日益先进的时代仍具挑战性,尤其当合成媒体愈发逼真时。本研究探索了当前最先进的多模态(推理)大语言模型(如OpenAI O1/4o、Gemini thinking Flash 2、Deepseek Janus、Grok 3、llama 3.2、Qwen 2/2.5 VL、Mistral Pixtral、Claude 3.5/3.7 sonnet)在深度伪造图像检测中的潜力。我们在多个数据集上对比12个最新多模态LLM与传统检测方法,包括近期发布的现实世界深度伪造图像。为提升性能,采用提示调优并深入分析模型推理路径,识别其决策关键因素。结果表明,最佳多模态LLM在零样本条件下表现具有竞争力,且在分布外数据集上甚至超越传统检测流水线;其余模型表现极差,部分低于随机猜测。此外,新版本及推理能力对这类特定任务无显著提升,而模型规模在某些情况下有助于性能提升。研究揭示了将多模态推理融入未来深度伪造检测框架的潜力,并提供了模型可解释性对真实场景鲁棒性的见解。
原文摘要 · Abstract (English)
Deepfake detection remains a critical challenge in the era of advanced generative models, particularly as synthetic media becomes more sophisticated. In this study, we explore the potential of state of the art multi-modal (reasoning) large language models (LLMs) for deepfake image detection such as (OpenAI O1/4o, Gemini thinking Flash 2, Deepseek Janus, Grok 3, llama 3.2, Qwen 2/2.5 VL, Mistral Pixtral, Claude 3.5/3.7 sonnet) . We benchmark 12 latest multi-modal LLMs against traditional deepfake detection methods across multiple datasets, including recently published real-world deepfake imagery. To enhance performance, we employ prompt tuning and conduct an in-depth analysis of the models' reasoning pathways to identify key contributing factors in their decision-making process. Our findings indicate that best multi-modal LLMs achieve competitive performance with promising generalization ability with zero shot, even surpass traditional deepfake detection pipelines in out-of-distribution datasets while the rest of the LLM families performs extremely disappointing with some worse than random guess. Furthermore, we found newer model version and reasoning capabilities does not contribute to performance in such niche tasks of deepfake detection while model size do help in some cases. This study highlights the potential of integrating multi-modal reasoning in future deepfake detection frameworks and provides insights into model interpretability for robustness in real-world scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。