用概率分布建模情绪,提升多模态情感识别精度。
Emotion as a Distribution: Joint Valence-Arousal Probability Learning for Speaker-Independent Multimodal Emotion Recognition

- 将情绪建模为效价-唤醒平面的概率分布,而非单一标签。
- 在IEMOCAP数据集上达到73.0%准确率,优于传统Transformer基线3.0个百分点。
- 分布输出可还原情绪环形结构,且熵值反映判断模糊性。
人类情绪是连续且常混合的,但多数多模态识别器将其简化为单一硬标签。我们主张识别器应输出情感空间上的概率分布。本文提出一个文本+语音系统,其在分类决策外,还生成9×9的效价-唤醒平面上的概率矩阵,采用二维高斯软标签与KL散度/交叉熵目标进行训练,旨在提供辅助支持。评估严格:采用说话人独立的5折留一会话验证(旋转会话内验证),仅在保留会话上报告主指标。在固定编码器-融合-头架构下,对比了Transformer与状态空间模型(Mamba-1/2/3)在匹配深度和宽度下的表现,两种运行点(T≈550,T≈2750)。所提双头系统在三组种子下达到73.0%±0.3的未加权准确率(独立重跑:72.1%),比Transformer融合基线提升3.0个未加权准确点(95%会话自助置信区间[1.0,4.7];配对t检验与会话级自助检验均显著),但在这些长度下无延迟或内存优势。将约100万可训练前端替换为冻结的WavLM-Large特征(可学习层权重)后,同一架构提升至76.6%±1.3。预设控制确保结论可信:更简单的效价-唤醒辅助任务可复现分类提升,而专用回归头略优地追踪连续评分,因此该头的核心价值在于归一化情感分布本身。该分布能恢复情绪环形结构:其质心轨迹与效价和唤醒高度相关(CCC 0.66/0.66;主要体现类间结构,类内跟踪较弱),且熵值与分类标注者模糊性弱但稳定关联,而非维度扩散。
原文摘要 · Abstract (English)
Human emotion is graded and frequently mixed, yet most multimodal recognizers collapse it onto a single hard label. We argue the recognizer should instead expose a distribution over affective space. Our text+speech system, alongside its categorical decision, emits a $9\times9$ probability matrix over the Valence-Arousal plane, trained with a two-dimensional Gaussian soft target under a Kullback-Leibler/cross-entropy objective, aimed at counseling support. Evaluation is strict: speaker-independent 5-fold leave-one-session-out IEMOCAP with rotating-session inner validation, headline metrics only on the held-out session. Within one fixed encoder-fusion-head pipeline we compare Transformer and state-space (Mamba-1/2/3) backbones at matched depth and width, at two operating points ($T\approx550$, $T\approx2750$). The featured dual-head system reaches 73.0% $\pm$ 0.3 unweighted accuracy over three seeds (separate rerun: 72.1%), exceeding the Transformer fusion baseline by 3.0 UA points (95% session-bootstrap CI [1.0,4.7]; significant under paired t-test and session-level bootstrap), with no latency or memory advantage at these lengths; swapping the ~1M trainable front-end for frozen WavLM-Large features (learnable layer weights) lifts the same architecture to 76.6% $\pm$ 1.3. Pre-specified controls scope the claims honestly: simpler valence-arousal auxiliaries reproduce the classification lift within noise, and a dedicated regression head tracks the continuous ratings slightly better, so the head's specific value is the normalized affect distribution itself. That distribution recovers the circumplex: its center of mass tracks valence and arousal (CCC 0.66/0.66; predominantly between-class structure, weaker within-class tracking), and its entropy is weakly but consistently linked to categorical rater ambiguity, not dimensional spread.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。