arXiv:2601.14274cs.LGcs.AI2026-01被引 1

通过拆解与精炼多模态情感信号,提升对话情感识别的准确性与可解释性。

Divide and Refine: Enhancing Multimodal Representation and Explainability for Emotion Recognition in Conversation

  • 分阶段处理:先分解各模态的独特、冗余与协同信息,再针对性优化。
  • 在IEMOCAP和MELD数据集上,多种模型架构均实现显著性能提升。
  • 方法可插拔,适用于不同多模态框架,增强结果可解释性。

对话中的多模态情感识别(MERC)需要有效融合来自多个模态的信号,包括模态特有线索、跨模态共享信息以及仅在融合时才出现的交互作用。从信息论角度看,这对应于独特性、冗余性和协同性贡献。理想表示应兼顾三者,但实现平衡仍具挑战。现有对比学习与增强方法常忽视数据准备对这些成分的保护作用,直接对原始输入或融合嵌入施加增强会模糊模态独特性与跨模态信号的边界。为此,我们提出两阶段框架「分割与精炼」(Divide and Refine, DnR)。在分割阶段,各模态被显式分解为独特性、成对冗余性和协同性成分;在精炼阶段,设计特定目标提升各成分的信息量,同时保持其独立性。精炼后的表示可无缝接入多种多模态流水线。在IEMOCAP与MELD上的大量实验表明,DnR在多个MERC骨干模型上均取得一致提升,验证了显式拆解、精炼与重组多模态表示作为原则性策略的有效性。代码已开源。

原文摘要 · Abstract (English)

Multimodal emotion recognition in conversation (MERC) requires representations that effectively integrate signals from multiple modalities. These signals include modality-specific cues, information shared across modalities, and interactions that emerge only when modalities are combined. In information-theoretic terms, these correspond to \emph{unique}, \emph{redundant}, and \emph{synergistic} contributions. An ideal representation should leverage all three, yet achieving such balance remains challenging. Recent advances in contrastive learning and augmentation-based methods have made progress, but they often overlook the role of data preparation in preserving these components. In particular, applying augmentations directly to raw inputs or fused embeddings can blur the boundaries between modality-unique and cross-modal signals. To address this challenge, we propose a two-phase framework \emph{\textbf{D}ivide and \textbf{R}efine} (\textbf{DnR}). In the \textbf{Divide} phase, each modality is explicitly decomposed into uniqueness, pairwise redundancy, and synergy. In the \textbf{Refine} phase, tailored objectives enhance the informativeness of these components while maintaining their distinct roles. The refined representations are plug-and-play compatible with diverse multimodal pipelines. Extensive experiments on IEMOCAP and MELD demonstrate consistent improvements across multiple MERC backbones. These results highlight the effectiveness of explicitly dividing, refining, and recombining multimodal representations as a principled strategy for advancing emotion recognition. Our implementation is available at https://github.com/mattam301/DnR-WACV2026

情感识别多模态可解释性表示学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。