通过可学习的融合标记实现多模态情感分析深度融合,性能领先。
DeepMLF: Multimodal language model with learnable tokens for deep fusion in sentiment analysis
- 引入可学习融合标记,在多层中逐步整合语音与视觉信息。
- 在三个基准上达顶尖效果,最优融合深度为5-7层,仅需约20个融合标记。
- 适合研究多模态深度融合与大模型扩展性的学者使用。
尽管多模态融合在多模态情感分析(MSA)中已广泛研究,但融合深度与多模态容量分配的作用仍不明确。本文将融合深度、可扩展性及专用多模态容量视为有效融合的关键因素。提出DeepMLF:一种具有可学习融合标记的新型多模态语言模型,专为深度融合设计。该模型结合音视频编码器与预训练解码语言模型,并在各层注入多模态信息。通过在语言模型中添加可学习标记,实现对模态交互的可控捕捉与各模态独立信息流的保留。这些融合标记通过因果自注意力在语言模型块中聚合语言信息,再经跨注意力在多模态块中融合音视频特征。作为专用多模态容量,该设计支持多层渐进式融合,实现融合过程的深度化。训练采用模态特定损失与语言建模损失联合优化,解码器需预测真实情感极性。在三个具有不同数据特性的MSA基准上,DeepMLF均达到当前最优性能。结果表明更深的融合带来更好表现,最优融合深度(5-7层)超过现有方法。同时,融合标记数量分析显示,约20个标记即达最佳性能。通过音视频编码器初始化实验验证表示学习顺序(融合课程)的重要性。消融实验证明所提融合设计与门控机制的优势,并全面评估了DeepMLF向大模型的可扩展性及各项训练目标与嵌入正则化的影响。
原文摘要 · Abstract (English)
While multimodal fusion has been extensively studied in Multimodal Sentiment Analysis (MSA), the role of fusion depth and multimodal capacity allocation remains underexplored. In this work, we position fusion depth, scalability, and dedicated multimodal capacity as primary factors for effective fusion. We introduce DeepMLF, a novel multimodal language model (LM) with learnable tokens tailored toward deep fusion. DeepMLF leverages an audiovisual encoder and a pretrained decoder LM augmented with multimodal information across its layers. We append learnable tokens to the LM that: 1) capture modality interactions in a controlled fashion and 2) preserve independent information flow for each modality. These fusion tokens gather linguistic information via causal self-attention in LM Blocks and integrate with audiovisual information through cross-attention MM Blocks. Serving as dedicated multimodal capacity, this design enables progressive fusion across multiple layers, providing depth in the fusion process. Our training recipe combines modality-specific losses and language modelling loss, with the decoder LM tasked to predict ground truth polarity. Across three MSA benchmarks with varying dataset characteristics, DeepMLF achieves state-of-the-art performance. Our results confirm that deeper fusion leads to better performance, with optimal fusion depths (5-7) exceeding those of existing approaches. Additionally, our analysis on the number of fusion tokens reveals that small token sets ($\sim$20) achieve optimal performance. We examine the importance of representation learning order (fusion curriculum) through audiovisual encoder initialization experiments. Our ablation studies demonstrate the superiority of the proposed fusion design and gating while providing a holistic examination of DeepMLF's scalability to LLMs, and the impact of each training objective and embedding regularization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。