通过分析多模态矛盾信号,精准识别情绪矛盾与犹豫状态。
Conflict-Aware Multimodal Fusion for Ambivalence and Hesitancy Recognition
- 用模态间差异作为双向线索,捕捉言语与表情声音的冲突
- 在BAH数据集上达0.715宏平均F1,比基线高超10个百分点
- 适合临床心理评估、人机交互中微妙情绪识别场景
情绪矛盾与犹豫(A/H)是人在不同通道间表现出不一致信号的微妙情感状态——例如言语与面部或语调传达相反信息。自动识别此类状态对临床应用有价值,但机器难以处理,因关键线索存在于各模态间的差异。本文提出ConflictAwareAH,采用三个预训练编码器提取视频、音频和文本表征,通过计算模态嵌入间的逐元素绝对差值构建成对冲突特征,作为双向提示:大跨模态差异标志A/H,小差异则确认行为一致性并锚定非A/H类别。该设计克服了以文本为主方法过度检测A/H(高F1-AH)却难确认其不存在的问题,相比纯文本模型,本方法在无A/H类上的F1提升4.6点,并使两类性能差距减半。引入文本引导的晚期融合策略,在推理时结合纯文本辅助头,额外提升4.1点宏平均F1。在ABAW10挑战赛的BAH数据集上,本方法在标注测试集上达到0.694宏平均F1,私有排行榜上达0.715,优于已有多模态基线超10点,且仅需单张GPU训练不足25分钟。
原文摘要 · Abstract (English)
Ambivalence and hesitancy (A/H) are subtle affective states where a person shows conflicting signals through different channels -- saying one thing while their face or voice tells another story. Recognising these states automatically is valuable in clinical settings, but it is hard for machines because the key evidence lives in the \emph{disagreements} between what is said, how it sounds, and what the face shows. We present \textbf{ConflictAwareAH}, a multimodal framework built for this problem. Three pre-trained encoders extract video, audio, and text representations. Pairwise conflict features -- element-wise absolute differences between modality embeddings -- serve as \emph{bidirectional} cues: large cross-modal differences flag A/H, while small differences confirm behavioural consistency and anchor the negative class. This conflict-aware design addresses a key limitation of text-dominant approaches, which tend to over-detect A/H (high F1-AH) while struggling to confirm its absence: our multimodal model improves F1-NoAH by +4.6 points over text alone and halves the class-performance gap. A complementary \emph{text-guided late fusion} strategy blends a text-only auxiliary head with the full model at inference, adding +4.1 Macro F1. On the BAH dataset from the ABAW10 Ambivalence/Hesitancy Challenge, our method reaches \textbf{0.694 Macro F1} on the labelled test split and \textbf{0.715} on the private leaderboard, outperforming published multimodal baselines by over 10 points -- all on a single GPU in under 25 minutes of training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。