用视觉语言模型识别面部属性，准确率超传统方法
Exploring Vision Language Models for Facial Attribute Recognition: Emotion, Race, Gender, and Age
- 用多模型融合的视觉语言框架分析人脸性别、种族、年龄和情绪
- 自研模型FaceScanPaliGemma在四项任务上分别达81.1%、95.8%、80%、59.4%准确率
- FaceScanGPT仅靠提示词即可多任务识别多人属性，适合复杂场景应用
人脸识别中的种族、性别、年龄和情绪分析在安防、广告内容审核、情感分析及社会行为研究中具有广泛应用。由于人脸属性的复杂性，传统方法依赖卷积神经网络（CNN）和大规模标注图像训练，虽有效但仍有提升空间。本文探索使用生成式预训练模型如GPT、GEMINI、LLAVA、PaliGemma和Microsoft Florence2进行面部属性识别。在FairFace、AffectNet和UTKFace等数据集上评估表明，视觉语言模型表现优异甚至超越传统方法。本文提出微调后的PaliGemma模型——FaceScanPaliGemma，其在种族、性别、年龄组和情绪分类上的准确率分别为81.1%、95.8%、80%和59.4%，优于原版PaliGemma及其他先进方法。此外，提出FaceScanGPT（基于GPT-4o），通过精心设计的提示词，可在多主体图像中识别个体的发型、衣着颜色、姿态等属性，展现强大多任务处理能力。
原文摘要 · Abstract (English)
Technologies for recognizing facial attributes like race, gender, age, and emotion have several applications, such as surveillance, advertising content, sentiment analysis, and the study of demographic trends and social behaviors. Analyzing demographic characteristics based on images and analyzing facial expressions have several challenges due to the complexity of humans' facial attributes. Traditional approaches have employed CNNs and various other deep learning techniques, trained on extensive collections of labeled images. While these methods demonstrated effective performance, there remains potential for further enhancements. In this paper, we propose to utilize vision language models (VLMs) such as generative pre-trained transformer (GPT), GEMINI, large language and vision assistant (LLAVA), PaliGemma, and Microsoft Florence2 to recognize facial attributes such as race, gender, age, and emotion from images with human faces. Various datasets like FairFace, AffectNet, and UTKFace have been utilized to evaluate the solutions. The results show that VLMs are competitive if not superior to traditional techniques. Additionally, we propose "FaceScanPaliGemma"--a fine-tuned PaliGemma model--for race, gender, age, and emotion recognition. The results show an accuracy of 81.1%, 95.8%, 80%, and 59.4% for race, gender, age group, and emotion classification, respectively, outperforming pre-trained version of PaliGemma, other VLMs, and SotA methods. Finally, we propose "FaceScanGPT", which is a GPT-4o model to recognize the above attributes when several individuals are present in the image using a prompt engineered for a person with specific facial and/or physical attributes. The results underscore the superior multitasking capability of FaceScanGPT to detect the individual's attributes like hair cut, clothing color, postures, etc., using only a prompt to drive the detection and recognition tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。