arXiv:2506.10452cs.CVcs.CL2025-06被引 4

提出新框架应对模态缺失与分布外数据,提升情感识别鲁棒性。

Towards Robust Multimodal Emotion Recognition under Missing Modalities and Distribution Shifts

  • 通过自蒸馏与因果推断结合,增强多模态情感识别鲁棒性
  • 在随机模态缺失和分布外场景下均优于现有方法
  • 参数少、训练快,适合实际部署

多模态情感识别(MER)近年面临模态缺失与分布外(OOD)数据的双重挑战。现有方法常依赖特定模型或引入过多参数,实用性受限。本文提出新型鲁棒MER框架CIDer,引入新任务随机模态特征缺失(RMFM)以更广义定义模态缺失。CIDer包含两个核心组件:模型特定自蒸馏(MSSD)模块与模型无关因果推断(MACI)模块。MSSD通过跨低层特征、注意力图与高层表示的权值共享自蒸馏,提升RMFM下的鲁棒性;同时,词级自对齐注意力模块(WSAM)降低计算开销,多模态复合变压器(MCT)实现高效融合。针对OOD问题,MACI利用定制因果图,通过多模态因果模块(MCM)与细粒度反事实文本缓解标签与语言偏差。值得注意的是,MACI可独立提升泛化能力,仅增加少量参数。此外,本文还构建了新的重组版MER OOD数据集。实验表明,CIDer在RMFM与OOD场景下表现优异,参数更少、训练更快,优于当前最优方法。代码已开源:https://github.com/gw-zhong/CIDer。

原文摘要 · Abstract (English)

Recent advancements in Multimodal Emotion Recognition (MER) face challenges in addressing both modality missing and Out-Of-Distribution (OOD) data simultaneously. Existing methods often rely on specific models or introduce excessive parameters, which limits their practicality. To address these issues, we propose a novel robust MER framework, Causal Inference Distiller (CIDer), and introduce a new task, Random Modality Feature Missing (RMFM), to generalize the definition of modality missing. CIDer integrates two key components: a Model-Specific Self-Distillation (MSSD) module and a Model-Agnostic Causal Inference (MACI) module. MSSD enhances robustness under the RMFM task through a weight-sharing self-distillation approach applied across low-level features, attention maps, and high-level representations. Additionally, a Word-level Self-aligned Attention Module (WSAM) reduces computational complexity, while a Multimodal Composite Transformer (MCT) facilitates efficient multimodal fusion. To tackle OOD challenges, MACI employs a tailored causal graph to mitigate label and language biases using a Multimodal Causal Module (MCM) and fine-grained counterfactual texts. Notably, MACI can independently enhance OOD generalization with minimal additional parameters. Furthermore, we also introduce the new repartitioned MER OOD datasets. Experimental results demonstrate that CIDer achieves robust performance in both RMFM and OOD scenarios, with fewer parameters and faster training compared to state-of-the-art methods. The implementation of this work is publicly accessible at https://github.com/gw-zhong/CIDer.

情感识别多模态鲁棒性因果推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。