模型能识别带上下文的表情,并还原人脑的感知过程。
How Do You Perceive My Face? Recognizing Facial Expressions in Multi-Modal Context by Modeling Mental Representations
- 用变分自编码-生成对抗网络分离内容与上下文特征
- 提出新注意力机制,使特征随上下文动态调整
- 生成的表情与人类感知高度一致,适合心理计算研究
人类面部表情的感知依赖先验知识和多模态上下文(如语音语调、情感文本、身体姿态等),即使面对中性表情,也能产生情绪判断。受此启发,我们提出一种新方法,不仅能准确分类感知到的表情,还能合成人类在特定情境下观察面部时的心理表征。通过变分自编码-生成对抗网络学习内容与上下文的独立表示,并引入新型注意力机制实现上下文依赖的特征适配。适配后的表示用于分类并生成增强上下文的表情。在真人实验中验证,合成表情能有效逼近人类心理表征。在RAVDESS数据集上达到81.01%准确率,在MEAD数据集上达到79.34%。代码已开源。
原文摘要 · Abstract (English)
Facial expression perception in humans inherently relies on prior knowledge and contextual cues, contributing to efficient and flexible processing. For instance, multi-modal emotional context (such as voice color, affective text, body pose, etc.) can prompt people to perceive emotional expressions in objectively neutral faces. Drawing inspiration from this, we introduce a novel approach for facial expression classification that goes beyond simple classification tasks. Our model accurately classifies a perceived face and synthesizes the corresponding mental representation perceived by a human when observing a face in context. With this, our model offers visual insights into its internal decision-making process. We achieve this by learning two independent representations of content and context using a VAE-GAN architecture. Subsequently, we propose a novel attention mechanism for context-dependent feature adaptation. The adapted representation is used for classification and to generate a context-augmented expression. We evaluate synthesized expressions in a human study, showing that our model effectively produces approximations of human mental representations. We achieve State-of-the-Art classification accuracies of 81.01% on the RAVDESS dataset and 79.34% on the MEAD dataset. We make our code publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。