解决多模态情感识别中模态冲突问题，提升模型准确性。
TiCAL:Typicality-Based Consistency-Aware Learning for Multimodal Emotion Recognition
- 基于典型性评估动态判断样本一致性，融合伪单模态标签。
- 在双曲空间嵌入特征，更好区分细微情感差异。
- 在CMU-MOSEI等数据集上比当前最优方法提升2.6%。
多模态情感识别（MER）旨在通过视觉、听觉和文本等异构模态数据准确识别人类情绪状态。现有方法主要依赖统一情绪标签进行训练，常忽视一个关键挑战：同一样本内不同模态间存在情绪矛盾。本文提出一种新框架TiCAL（基于典型性的自洽学习），受人类情绪感知阶段特性的启发，通过伪单模态情绪标签与典型性估计动态评估每个训练样本的一致性。为增强情绪表征，将特征嵌入双曲空间，以捕捉情绪类别间的细微差异。通过将一致性估计融入学习过程，该方法显著提升了在高模态不一致样本上的表现。在标准数据集如CMU-MOSEI和MER2023上的大量实验表明，TiCAL有效缓解了模态间情绪冲突，整体识别准确率较当前最优方法DMD提升约2.6%。
原文摘要 · Abstract (English)
Multimodal Emotion Recognition (MER) aims to accurately identify human emotional states by integrating heterogeneous modalities such as visual, auditory, and textual data. Existing approaches predominantly rely on unified emotion labels to supervise model training, often overlooking a critical challenge: inter-modal emotion conflicts, wherein different modalities within the same sample may express divergent emotional tendencies. In this work, we address this overlooked issue by proposing a novel framework, Typicality-based Consistent-aware Multimodal Emotion Recognition (TiCAL), inspired by the stage-wise nature of human emotion perception. TiCAL dynamically assesses the consistency of each training sample by leveraging pseudo unimodal emotion labels alongside a typicality estimation. To further enhance emotion representation, we embed features in a hyperbolic space, enabling the capture of fine-grained distinctions among emotional categories. By incorporating consistency estimates into the learning process, our method improves model performance, particularly on samples exhibiting high modality inconsistency. Extensive experiments on benchmark datasets, e.g, CMU-MOSEI and MER2023, validate the effectiveness of TiCAL in mitigating inter-modal emotional conflicts and enhancing overall recognition accuracy, e.g., with about 2.6% improvements over the state-of-the-art DMD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。