arXiv:2608.30563cs.CV2026-08中稿 · EMNLP

针对多模态情感识别中缺失模态问题,提出分拆记忆蒸馏方法提升鲁棒性。

Modality Disentangled Learning for Incomplete Multimodal Emotion Recognition: A Primitive Memory Distillation Perspective

论文配图:Modality Disentangled Learning for Incomplete Multimodal Emotion Recognition: A Primitive Memory Distillation Perspective
图 1 · 摘自论文原文
  • 将模态内信息按可恢复性分拆为可学习语义单元,构建特定记忆库
  • 在缺失模态时,通过共享语义动态检索记忆单元,补偿特定信息
  • 在IEMOCAP、CMU-MOSI等数据集上实现最优性能与强鲁棒性

多模态情感识别系统在真实场景中常面临模态缺失问题。现有方法通常整体生成、对齐或蒸馏缺失模态,忽视了各模态携带信息的异质性。这种整体处理方式混淆了可推断的共享语义与不确定的模态特异性细节,导致表示不稳定、鲁棒性下降。为此,我们提出原始记忆蒸馏(PriMD)框架。不同于以往方法,PriMD采用模态内视角,关注模态内部不同信息类型的可恢复性差异。首先,将跨模态共享语义与模态特异性表示解耦;其次,将后者离散化为可学习的语义原型,构建模态特异性记忆库。当模态缺失时,该框架以可用模态的共享语义作为查询,动态检索原型,于受限记忆空间内补偿缺失的模态特异性信息,并与教师模型对齐。在IEMOCAP、CMU-MOSI和CMU-MOSEI上的大量实验表明,PriMD在多种缺失模态设置下均达到当前最佳性能,且表现出更强的一致性鲁棒性,有效缓解了整体特征推理带来的不稳定性。代码与项目主页见https://github.com/JiaqiZhang-Sengoku/PriMD 和 https://jiaqizhang-sengoku.github.io/PriMD/

原文摘要 · Abstract (English)

Multimodal Emotion Recognition (MER) systems often suffer from missing modalities in real-world scenarios. Existing methods usually generate, align, or distill missing modalities as a whole, overlooking the heterogeneous nature of the information carried by each modality. Such holistic treatment mixes inferable shared semantics with uncertain modality-specific details, yielding unstable representations and degrading robustness. To address this issue, we propose the Primitive Memory Distillation (PriMD) framework. Unlike existing methods, PriMD takes an intra-modal perspective and focuses on how different types of information within a modality differ in recoverability within each modality. PriMD first disentangles cross-modal shared semantics from modality-specific representations, and then discretizes the latter into learnable semantic primitives to construct modality-specific memory banks. When modalities are missing, PriMD is a teacher-student framework that the student model uses the shared semantics of available modalities as queries to dynamically retrieve primitives. It compensates for missing modality-specific information within a constrained memory space and aligns with the teacher model. Extensive experiments on IEMOCAP, CMU-MOSI, and CMU-MOSEI demonstrate that PriMD achieves state-of-the-art performance and consistently stronger robustness across a wide range of missing-modality settings, while mitigating the instability caused by holistic feature inference. Our code and project website are available at https://github.com/JiaqiZhang-Sengoku/PriMD and https://jiaqizhang-sengoku.github.io/PriMD/, respectively.

情感识别多模态缺失数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。