用大模型检测人脸欺骗攻击,跨数据集表现更优。
Foundation and Multimodal Large Language Models for Face Presentation and Morph Attack Detection
- 用大模型提取人脸图像中的欺骗特征,支持文本推理
- 微调后模型在跨数据集测试中达到最新最好性能
- 适合安全认证系统开发者参考使用
人脸识别系统广泛应用于关键安全场景,但仍易受呈现攻击和变脸攻击威胁。呈现攻击检测(PAD)与变脸攻击检测(MAD)是保障可信生物识别的关键。尽管已有进展,现有检测器泛化能力有限,在跨数据集评估中性能下降。本文系统研究通用基础模型(FMs)与多模态大语言模型(MLLMs)是否蕴含PAD与MAD相关信息,以及如何最优部署。考察五种方法:(i)直接调用现成MLLM进行零样本提示;(ii)在MLLM输出词元概率上训练浅层模型;(iii)在任务问答数据上参数高效微调,生成专用模型PADLLM与MADLLM,可提供决策文本解释;(iv)冻结视觉编码器的线性探测;(v)微调FMs与MLLMs的视觉编码器。在四个PAD数据集(MSU-MFSD、CASIA-FASD、Replay-Attack、OULU-NPU)和四个MAD数据集(FFHQ、FRGC、FRLL、FERET)上评测16个开源多模态大模型与30个视觉编码器。结果表明,基础模型与多模态大模型在两类任务中均取得显著性能。尤其是微调模型在跨数据集评估中达到最先进水平,说明通用预训练表征包含丰富攻击相关线索。所有实验代码将公开发布。
原文摘要 · Abstract (English)
Face recognition systems are increasingly deployed in security-critical applications, yet they remain vulnerable to presentation and morph attacks. Presentation attack detection (PAD) and morphing attack detection (MAD) are therefore essential components of trustworthy face biometrics. Despite advancements in PAD and MAD methods, existing detectors suffer from limited generalization and degrade in cross-dataset evaluation. In this paper, we systematically investigate whether general-purpose foundation models (FMs) and multimodal large language models (MLLMs) encode PAD-relevant and MAD-relevant information, and how such models can best be deployed for both tasks. We study five approaches with increasing access to the internal information of the model: (i) zero-shot prompting of off-the-shelf MLLMs; (ii) training a shallow model on the next-token logit probabilities at the output of the MLLM; (iii) parameter-efficient fine-tuning on task-specific question-answer data, yielding two specialized MLLMs, called PADLLM and MADLLM, which additionally provide textual reasoning for their decisions; (iv) linear probing of frozen vision encoders; and (v) fine-tuning of vision encoders of FMs and MLLMs. We benchmark 16 open-weight MLLMs and 30 vision encoder backbones on four PAD datasets (MSU-MFSD, CASIA-FASD, Replay-Attack, and OULU-NPU) and four MAD datasets (FFHQ, FRGC, FRLL, and FERET). Our experiments show that FMs and MLLMs can achieve significant performance for PAD and MAD. In addition, the fine-tuned models achieve state-of-the-art detection performance in cross-dataset evaluation, indicating that general-purpose pretrained representations carry substantial attack-relevant information. Source code of all our experiments will be publicly released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。