用多模态大模型实现可解释的面部反欺骗,能判断真假、识别攻击类型并定位区域。
FaceShield: Explainable Face Anti-Spoofing with Multimodal Large Language Models
- 融合原图与先验信息,通过提示引导的视觉标记掩码提升泛化能力。
- 在四个任务上超越现有深度学习模型和通用多模态大模型,最高准确率达98.7%。
- 适合安全防护、可信认证场景,尤其需要可解释性的应用。
面部反欺骗(FAS)对保护人脸识别系统免受呈现攻击至关重要。以往方法将此任务视为分类问题,缺乏预测结果的可解释性与推理过程。近期多模态大语言模型(MLLM)在视觉感知、推理与决策方面展现出强大能力。然而,目前尚无专为FAS任务设计的通用且全面的MLLM及数据集。为此,我们提出FaceShield,一种用于FAS的MLLM,以及相应的预训练与监督微调(SFT)数据集:FaceShield-pre10K和FaceShield-sft45K。FaceShield可判定人脸真伪、识别攻击类型、提供判断理由并定位攻击区域。具体而言,我们采用基于先验知识的欺骗感知视觉感知(SAVP),结合原始图像与辅助信息;再使用提示引导的视觉标记掩码(PVTM)随机遮蔽视觉标记,提升模型泛化能力。我们在三个基准数据集上进行了广泛实验,结果表明,FaceShield在四项FAS任务——粗粒度分类、细粒度分类、推理与攻击定位——上显著优于先前深度学习模型及通用MLLM。我们的指令数据集、协议与代码将公开于https://github.com/Why0912/FaceShield。
原文摘要 · Abstract (English)
Face anti-spoofing (FAS) is crucial for protecting facial recognition systems from presentation attacks. Previous methods approached this task as a classification problem, lacking interpretability and reasoning behind the predicted results. Recently, multimodal large language models (MLLMs) have shown strong capabilities in perception, reasoning, and decision-making in visual tasks. However, there is currently no universal and comprehensive MLLM and dataset specifically designed for FAS task. To address this gap, we propose FaceShield, a MLLM for FAS, along with the corresponding pre-training and supervised fine-tuning (SFT) datasets, FaceShield-pre10K and FaceShield-sft45K. FaceShield is capable of determining the authenticity of faces, identifying types of spoofing attacks, providing reasoning for its judgments, and detecting attack areas. Specifically, we employ spoof-aware vision perception (SAVP) that incorporates both the original image and auxiliary information based on prior knowledge. We then use an prompt-guided vision token masking (PVTM) strategy to random mask vision tokens, thereby improving the model's generalization ability. We conducted extensive experiments on three benchmark datasets, demonstrating that FaceShield significantly outperforms previous deep learning models and general MLLMs on four FAS tasks, i.e., coarse-grained classification, fine-grained classification, reasoning, and attack localization. Our instruction datasets, protocols, and codes will be released at https://github.com/Why0912/FaceShield.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。