针对多模态情感推理中的情绪冲突问题,提出新基准与平衡融合框架。
Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoning
- 构建包含三类情境的CA-MER基准,检验模型在情绪冲突下的表现。
- 现有模型过度依赖音频,视觉线索常被忽略,尤其在冲突场景下。
- 提出MoSEAR框架,实现音频与视觉模态的均衡融合,适合多模态系统优化者。
尽管现有多模态大语言模型在多模态情感推理中表现优异,但常忽视情感冲突场景——即不同模态间情绪线索不一致的情况。为此,我们首先提出CA-MER基准,用于评估多模态大模型在真实情感冲突下的能力,包含三个子集:视频对齐、音频对齐和一致样本,其中仅一个或所有模态反映真实情绪。然而,对CA-MER的评估显示,当前最先进的情感推理模型在冲突情境下系统性地过度依赖音频信号,忽视了视觉模态的关键线索。为缓解此偏差,我们提出参数高效框架MoSEAR,促进模态间均衡融合。该框架包含两个模块:(1) MoSE,具有正则化门控机制的模态专用专家,降低微调头中的模态偏见;(2) AR,注意力重分配机制,在推理时重新平衡冻结主干中的模态贡献。该框架能有效缓解情绪冲突,同时提升一致样本表现,且不产生音视频模态间的性能权衡。在多个基准(包括MER2023、EMER、DFEW和我们的CA-MER)上的实验表明,MoSEAR在模态冲突条件下达到最先进水平。
原文摘要 · Abstract (English)
Despite their strong performance in multimodal emotion reasoning, existing Multimodal Large Language Models (MLLMs) often overlook the scenarios involving emotion conflicts, where emotional cues from different modalities are inconsistent. To fill this gap, we first introduce CA-MER, a new benchmark designed to examine MLLMs under realistic emotion conflicts. It consists of three subsets: video-aligned, audio-aligned, and consistent, where only one or all modalities reflect the true emotion. However, evaluations on our CA-MER reveal that current state-of-the-art emotion MLLMs systematically over-rely on audio signal during emotion conflicts, neglecting critical cues from visual modality. To mitigate this bias, we propose MoSEAR, a parameter-efficient framework that promotes balanced modality integration. MoSEAR consists of two modules: (1)MoSE, modality-specific experts with a regularized gating mechanism that reduces modality bias in the fine-tuning heads; and (2)AR, an attention reallocation mechanism that rebalances modality contributions in frozen backbones during inference. Our framework offers two key advantages: it mitigates emotion conflicts and improves performance on consistent samples-without incurring a trade-off between audio and visual modalities. Experiments on multiple benchmarks-including MER2023, EMER, DFEW, and our CA-MER-demonstrate that MoSEAR achieves state-of-the-art performance, particularly under modality conflict conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。