arXiv:2602.09121cs.AI2026-02被引 2

用狄利克雷证据建模多模态情绪识别的不确定性,适合边缘设备部署。

Uncertainty-Aware Multimodal Emotion Recognition through Dirichlet Parameterization

  • 基于狄利克雷证据的融合机制,无需额外训练即可捕捉预测不确定性。
  • 在5个基准数据集上表现接近最优,且对缺失或模糊输入保持鲁棒。
  • 模块化设计支持扩展其他模态,适用于医疗与人机交互场景。

本文提出一种轻量级、隐私友好的多模态情绪识别(MER)框架,专为边缘设备部署设计。系统采用语音、文本和面部图像三种模态,支持灵活扩展。各模态分别使用Emotion2Vec(语音)、基于ResNet的模型(面部表情)和DistilRoBERTa(文本)作为高效推理骨干网络。为融合跨模态不确定性,引入基于德普斯特-沙弗理论与狄利克雷证据的模型无关融合机制,直接作用于模型原始输出 logits,无需额外训练或联合分布估计,具备广泛适用性。在eNTERFACE05、MEAD、MELD、RAVDESS 和 CREMA-D 五个基准数据集上的验证表明,该方法在保持计算高效的同时,实现竞争性准确率,并对模糊或缺失输入具有鲁棒性。整体框架强调模块化、可扩展性与实际可行性,为医疗、人机交互等情绪感知应用中的不确定性感知系统提供新路径。

原文摘要 · Abstract (English)

In this work, we present a lightweight and privacy-preserving Multimodal Emotion Recognition (MER) framework designed for deployment on edge devices. To demonstrate framework's versatility, our implementation uses three modalities - speech, text and facial imagery. However, the system is fully modular, and can be extended to support other modalities or tasks. Each modality is processed through a dedicated backbone optimized for inference efficiency: Emotion2Vec for speech, a ResNet-based model for facial expressions, and DistilRoBERTa for text. To reconcile uncertainty across modalities, we introduce a model- and task-agnostic fusion mechanism grounded in Dempster-Shafer theory and Dirichlet evidence. Operating directly on model logits, this approach captures predictive uncertainty without requiring additional training or joint distribution estimation, making it broadly applicable beyond emotion recognition. Validation on five benchmark datasets (eNTERFACE05, MEAD, MELD, RAVDESS and CREMA-D) show that our method achieves competitive accuracy while remaining computationally efficient and robust to ambiguous or missing inputs. Overall, the proposed framework emphasizes modularity, scalability, and real-world feasibility, paving the way toward uncertainty-aware multimodal systems for healthcare, human-computer interaction, and other emotion-informed applications.

情绪识别多模态不确定性建模边缘计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。