让多模态大模型更准地识破合成图像,还能解释为什么这么判断。
DF-LLaVA: Unlocking MLLMs for Synthetic Image Detection via Knowledge Injection and Conflict-Driven Self-Reflection
- 从大模型自身挖掘知识并注入,提升判别能力。
- 推理时通过冲突信号触发自我反思,准确率超越专家模型。
- 适合需要高精度+可解释性合成图像检测的场景。
随着合成图像泛滥,如何在保持人类可理解性的前提下精准评估图像真实性并定位伪造内容,仍是难题。现有检测模型多仅提供真伪概率或二元判断,解释力有限。尽管基于多模态大模型(MLLM)的方法更具可解释性,但在纯真实性分类准确率上仍落后于专家模型。为此,我们提出DF-LLaVA,一种全新有效框架,激发MLLM内在判别潜力。该方法首先从LLaVA模型中挖掘隐含知识,并通过微调注入;推理时,由预测冲突信号触发自我反思机制,生成最终优化响应。实验表明,该框架使LLaVA在合成图像检测上达到超越专家模型的准确率,同时保留了MLLM的可解释性优势。代码已开源:https://github.com/Eliot-Shen/DF-LLaVA。
原文摘要 · Abstract (English)
With the increasing prevalence of synthetic images, evaluating image authenticity and locating forgeries accurately while maintaining human interpretability remains a challenging task. Existing detection models primarily focus on simple authenticity classification, ultimately providing only a forgery probability or binary judgment, which offers limited explanatory insights into image authenticity. Moreover, while MLLM-based detection methods can provide more interpretable results, they still lag behind expert models in terms of pure authenticity classification accuracy. To address this, we propose DF-LLaVA, a novel and effective framework that unlocks the intrinsic discrimination potential of MLLMs. Our approach first mines latent knowledge from the MLLM itself and then injects it into the model via fine-tuning. During inference, conflict signals arising from the model's predictions activate a self-reflection process, leading to the final refined responses. This framework allows LLaVA to achieve outstanding detection accuracy exceeding expert models while still maintaining the interpretability offered by MLLMs. Extensive experiments confirm the superiority of DF-LLaVA, achieving both high accuracy and explainability in synthetic image detection. Code is available online at: https://github.com/Eliot-Shen/DF-LLaVA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。