解决语音情绪识别中语义与音调冲突问题，提升真实场景下识别准确率。
When Tone and Words Disagree: Towards Robust Speech Emotion Recognition under Acoustic-Semantic Conflict
- 分离声学与语义路径，用轻量注意力模块桥接两者。
- 在新构建的CASE数据集上达到59.38%准确率，显著优于现有模型。
- 适合需要应对真实对话中情绪矛盾的应用场景。
语音情绪识别（SER）系统通常假设声学情绪与词汇语义一致，但在真实交互中，声学-语义冲突普遍存在——语气传达的情绪与话语字面意义相悖。我们发现，当前最先进的SER模型，包括基于ASR、自监督学习（SSL）和音频语言模型（ALM）的方法，在此类冲突下性能显著下降，原因在于语义偏见或声学-语义表征纠缠。为此，我们提出融合声学-语义（FAS）框架，显式解耦声学与语义路径，并通过轻量级查询注意力模块进行连接。为实现系统评估，我们构建了首个以清晰可解释声学-语义冲突为主的数据集CASE，覆盖多样场景。大量实验表明，FAS在域内及零样本设置下均优于现有方法。尤其在CASE基准上，传统模型表现极差，而FAS以59.38%准确率刷新SOTA。代码与数据已开源。
原文摘要 · Abstract (English)
Speech Emotion Recognition (SER) systems often assume congruence between vocal emotion and lexical semantics. However, in real-world interactions, acoustic-semantic conflict is common yet overlooked, where the emotion conveyed by tone contradicts the literal meaning of spoken words. We show that state-of-the-art SER models, including ASR-based, self-supervised learning (SSL) approaches and Audio Language Models (ALMs), suffer performance degradation under such conflicts due to semantic bias or entangled acoustic-semantic representations. To address this, we propose the Fusion Acoustic-Semantic (FAS) framework, which explicitly disentangles acoustic and semantic pathways and bridges them through a lightweight, query-based attention module. To enable systematic evaluation, we introduce the Conflict in Acoustic-Semantic Emotion (CASE), the first dataset dominated by clear and interpretable acoustic-semantic conflicts in varied scenarios. Extensive experiments demonstrate that FAS consistently outperforms existing methods in both in-domain and zero-shot settings. Notably, on the CASE benchmark, conventional SER models fail dramatically, while FAS sets a new SOTA with 59.38% accuracy. Our code and datasets is available at https://github.com/24DavidHuang/FAS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。