用大模型生成的推理路径提升多模态情感识别准确率
Multimodal Video Emotion Recognition with Reliable Reasoning Priors
- 用Gemini生成细粒度跨模态推理轨迹作为先验知识
- 在MER2024上实现显著性能提升，验证了先验可靠性
- 适合关注多模态情感分析与大模型融合的研究者
本研究探索将多模态大模型（MLLM）生成的可信推理先验知识融入多模态情感识别。我们采用Gemini生成细粒度、模态可分离的推理轨迹，并在融合阶段注入作为先验，增强跨模态交互。为缓解多模态情感识别中显著的类别不平衡问题，提出平衡双对比学习（Balanced Dual-Contrastive Learning），联合优化类间与类内分布。在MER2024基准测试上，该先验增强框架取得显著性能提升，表明MLLM衍生推理的可靠性可与轻量级融合网络的领域适应性协同作用，实现鲁棒且可扩展的情感识别。
原文摘要 · Abstract (English)
This study investigates the integration of trustworthy prior reasoning knowledge from MLLMs into multimodal emotion recognition. We employ Gemini to generate fine-grained, modality-separable reasoning traces, which are injected as priors during the fusion stage to enrich cross-modal interactions. To mitigate the pronounced class-imbalance in multimodal emotion recognition, we introduce Balanced Dual-Contrastive Learning, a loss formulation that jointly balances inter-class and intra-class distributions. Applied to the MER2024 benchmark, our prior-enhanced framework yields substantial performance gains, demonstrating that the reliability of MLLM-derived reasoning can be synergistically combined with the domain adaptability of lightweight fusion networks for robust, scalable emotion recognition.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。