用伪造掩码生成精准描述,提升人脸伪造检测的准确性。
Towards General Visual-Linguistic Face Forgery Detection(V2)
- 通过伪造掩码定位区域与类型,减少大模型幻觉
- 在多个基准上提升检测性能,区域识别准确率更高
- 适合需要高可靠文本标注的伪造检测研究者
人脸篡改技术快速发展,严重威胁安全与社会信任。现有方法虽利用多模态模型提升检测泛化性与可解释性,但人工标注或直接由多模态大模型生成的文本描述常存在幻觉,尤其在高质量伪造图像上表现不佳。为此,我们提出人脸伪造文本生成器(FFTG),一种新型标注流程:先利用伪造掩码进行区域与类型初步识别,再通过系统化提示策略引导多模态大模型降低幻觉。我们通过微调CLIP(采用三分支训练框架,融合单模态与多模态目标)和多模态大模型,验证该方法的有效性。实验表明,该方法不仅显著提升标注准确性,区域识别准确率更高,还在多个伪造检测基准上实现性能提升。代码已开源:https://github.com/skJack/VLFFD.git。
原文摘要 · Abstract (English)
Face manipulation techniques have achieved significant advances, presenting serious challenges to security and social trust. Recent works demonstrate that leveraging multimodal models can enhance the generalization and interpretability of face forgery detection. However, existing annotation approaches, whether through human labeling or direct Multimodal Large Language Model (MLLM) generation, often suffer from hallucination issues, leading to inaccurate text descriptions, especially for high-quality forgeries. To address this, we propose Face Forgery Text Generator (FFTG), a novel annotation pipeline that generates accurate text descriptions by leveraging forgery masks for initial region and type identification, followed by a comprehensive prompting strategy to guide MLLMs in reducing hallucination. We validate our approach through fine-tuning both CLIP with a three-branch training framework combining unimodal and multimodal objectives, and MLLMs with our structured annotations. Experimental results demonstrate that our method not only achieves more accurate annotations with higher region identification accuracy, but also leads to improvements in model performance across various forgery detection benchmarks. Our Codes are available in https://github.com/skJack/VLFFD.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。