arXiv:2507.02205cs.CV2025-07被引 3

零样本多模态融合,识别复杂情绪组合,无需特定训练数据。

Team RAS in 9th ABAW Competition: Multimodal Compound Expression Recognition Approach

  • 六模态融合:面部、场景、音频、文本等,通过零样本模型统一处理。
  • 零样本测试下在三个数据集上F1达46.95%~49.02%,媲美有监督方法。
  • 动态加权融合与概率聚合设计,输出可解释的复合情绪结果,适合跨域应用。

复合表情识别(CER)是情感计算的一个分支,旨在检测由基本情绪组合形成的复杂情感状态。本文提出一种新颖的零样本多模态方法,将静态与动态面部表情、场景与标签匹配、场景上下文、音频和文本共六种异构模态整合至统一流程中。不同于依赖任务特定训练数据的现有方法,本方法采用零样本组件,包括基于CLIP的标签匹配与Qwen-VL实现的语义场景理解。进一步引入多头概率融合(MHPF)模块,动态加权各模态预测,并通过成对概率聚合(PPA)与成对特征相似性聚合(PFSA)构建复合表情(CE)转换模块,生成可解释的复合情绪输出。在多语料库训练下,该方法在AffWild2、AFEW与C-EXPR-DB上的零样本测试F1分数分别为46.95%、49.02%与34.85%,表现接近在目标数据上训练的监督模型,证明了其无需领域适配即可有效捕捉复合表情的能力。源代码已公开。

原文摘要 · Abstract (English)

Compound Expression Recognition (CER), a subfield of affective computing, aims to detect complex emotional states formed by combinations of basic emotions. In this work, we present a novel zero-shot multimodal approach for CER that combines six heterogeneous modalities into a single pipeline: static and dynamic facial expressions, scene and label matching, scene context, audio, and text. Unlike previous approaches relying on task-specific training data, our approach uses zero-shot components, including Contrastive Language-Image Pretraining (CLIP)-based label matching and Qwen-VL for semantic scene understanding. We further introduce a Multi-Head Probability Fusion (MHPF) module that dynamically weights modality-specific predictions, followed by a Compound Expressions (CE) transformation module that uses Pair-Wise Probability Aggregation (PPA) and Pair-Wise Feature Similarity Aggregation (PFSA) methods to produce interpretable compound emotion outputs. Evaluated under multi-corpus training, the proposed approach shows F1 scores of 46.95% on AffWild2, 49.02% on Acted Facial Expressions in The Wild (AFEW), and 34.85% on C-EXPR-DB via zero-shot testing, which is comparable to the results of supervised approaches trained on target data. This demonstrates the effectiveness of the proposed approach for capturing CE without domain adaptation. The source code is publicly available.

情绪识别多模态零样本复合表情

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。