提出可解释时序对齐的多模态情感分析模型，提升细粒度情绪识别效果。
Text-Routed Sparse Mixture-of-Experts Model with Explanation and Temporal Alignment for Multi-Modal Sentiment Analysis
- 用文本路由稀疏专家模型，结合解释生成与时序对齐机制
- 在四数据集上表现最优，CH-SIMS上误差降低13.5%至0.353
- 适合需要可解释性和时序敏感性的多模态情感分析场景
涉及人类交互的应用亟需多模态情感分析（MSA）。尽管已有多种方法处理不同模态中的细微情绪，但解释能力与时序对齐的潜力仍待挖掘。本文提出文本路由稀疏专家模型，结合解释与时间对齐（TEXT）。TEXT首先通过多模态大语言模型增强解释生成，再通过面向时序的神经网络模块对齐音视频表征。该模型融合门控融合机制，创新性地结合Mamba与时间交叉注意力优势。实验显示，TEXT在四个数据集上均优于所有对比模型，包括三种近期方法和三种多模态大语言模型，在六项指标中至少有四项领先。例如在CH-SIMS数据集上，平均绝对误差降至0.353，较最近方法降低13.5%。
原文摘要 · Abstract (English)
Human-interaction-involved applications underscore the need for Multi-modal Sentiment Analysis (MSA). Although many approaches have been proposed to address the subtle emotions in different modalities, the power of explanations and temporal alignments is still underexplored. Thus, this paper proposes the Text-routed sparse mixture-of-Experts model with eXplanation and Temporal alignment for MSA (TEXT). TEXT first augments explanations for MSA via Multi-modal Large Language Models (MLLM), and then novelly aligns the epresentations of audio and video through a temporality-oriented neural network block. TEXT aligns different modalities with explanations and facilitates a new text-routed sparse mixture-of-experts with gate fusion. Our temporal alignment block merges the benefits of Mamba and temporal cross-attention. As a result, TEXT achieves the best performance cross four datasets among all tested models, including three recently proposed approaches and three MLLMs. TEXT wins on at least four metrics out of all six metrics. For example, TEXT decreases the mean absolute error to 0.353 on the CH-SIMS dataset, which signifies a 13.5% decrement compared with recently proposed approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。