针对多模态情感识别中缺失模态问题,提出动态难易感知训练框架。
Hardness-Aware Dynamic Curriculum Learning for Robust Multimodal Emotion Recognition with Missing Modalities
- 根据直接与间接难度评估样本难易程度
- 通过检索机制动态调整训练重点,提升对难题的处理能力
- 适合研究多模态学习与鲁棒性建模的读者
缺失模态已成为多模态情感识别(MER)的关键研究方向。传统方法通常通过重建缺失模态来应对,但未能考虑不同样本间重建难度的差异,导致模型对困难样本处理能力不足。为此,本文提出一种全新的难易感知动态课程学习框架HARDY-MER。该框架包含两个关键阶段:首先估计每个样本的难易程度,其次在训练中策略性地强调困难样本以提升模型对这些挑战性实例的表现。具体而言,我们引入多视图难易度评估机制,通过直接难度(模态重建误差)和间接难度(跨模态互信息)量化重建复杂度;同时提出基于检索的动态课程学习策略,通过检索语义相似样本并平衡易/难样本的学习权重,实现训练过程的自适应调整。在多个基准数据集上的大量实验表明,HARDY-MER在缺失模态场景下持续优于现有方法。代码将公开于 https://github.com/HARDY-MER/HARDY-MER。
原文摘要 · Abstract (English)
Missing modalities have recently emerged as a critical research direction in multimodal emotion recognition (MER). Conventional approaches typically address this issue through missing modality reconstruction. However, these methods fail to account for variations in reconstruction difficulty across different samples, consequently limiting the model's ability to handle hard samples effectively. To overcome this limitation, we propose a novel Hardness-Aware Dynamic Curriculum Learning framework, termed HARDY-MER. Our framework operates in two key stages: first, it estimates the hardness level of each sample, and second, it strategically emphasizes hard samples during training to enhance model performance on these challenging instances. Specifically, we first introduce a Multi-view Hardness Evaluation mechanism that quantifies reconstruction difficulty by considering both Direct Hardness (modality reconstruction errors) and Indirect Hardness (cross-modal mutual information). Meanwhile, we introduce a Retrieval-based Dynamic Curriculum Learning strategy that dynamically adjusts the training curriculum by retrieving samples with similar semantic information and balancing the learning focus between easy and hard instances. Extensive experiments on benchmark datasets demonstrate that HARDY-MER consistently outperforms existing methods in missing-modality scenarios. Our code will be made publicly available at https://github.com/HARDY-MER/HARDY-MER.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。