提出双向时序蒸馏网络，解决多模态情感分析中缺失数据问题。
Modality-Invariant Bidirectional Temporal Representation Distillation Network for Missing Multimodal Sentiment Analysis
- 用完整模态指导缺失模态的蒸馏学习，提升鲁棒性。
- 引入双向时序表征模块，缓解多模态数据异构性。
- 适合处理真实场景下不完整多模态数据的情感分析任务。
多模态情感分析（MSA）融合文本、音频和视频等不同模态，全面理解个体情绪状态。然而，现实数据中不完整模态的随机缺失带来了显著挑战。此外，多模态数据的异构性尚未得到有效解决。为此，我们提出针对缺失多模态情感分析的模态无关双向时序表征蒸馏网络（MITR-DNet）。该方法通过蒸馏机制，由完整模态的教师模型指导缺失模态的学生模型，增强在模态缺失情况下的鲁棒性。同时，设计了模态无关双向时序表征学习模块（MIB-TRL），有效缓解多模态数据的异构性问题。
原文摘要 · Abstract (English)
Multimodal Sentiment Analysis (MSA) integrates diverse modalities(text, audio, and video) to comprehensively analyze and understand individuals' emotional states. However, the real-world prevalence of incomplete data poses significant challenges to MSA, mainly due to the randomness of modality missing. Moreover, the heterogeneity issue in multimodal data has yet to be effectively addressed. To tackle these challenges, we introduce the Modality-Invariant Bidirectional Temporal Representation Distillation Network (MITR-DNet) for Missing Multimodal Sentiment Analysis. MITR-DNet employs a distillation approach, wherein a complete modality teacher model guides a missing modality student model, ensuring robustness in the presence of modality missing. Simultaneously, we developed the Modality-Invariant Bidirectional Temporal Representation Learning Module (MIB-TRL) to mitigate heterogeneity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。