提出双流对齐与分层瓶颈融合框架,提升多模态情感分析性能
DashFusion: Dual-stream Alignment with Hierarchical Bottleneck Fusion for Multimodal Sentiment Analysis
- 通过时序与语义双通道对齐,实现跨模态特征同步
- 在三个数据集上达到最新最优效果,尤其在CMU-MOSEI上提升显著
- 适合需要高效多模态融合的场景,如智能客服、情感计算
多模态情感分析(MSA)融合文本、图像和音频等多种模态信息,以更全面地理解情感。然而,有效MSA面临对齐与融合的挑战:对齐需同步各模态的时间与语义信息,融合则需将对齐后的特征整合为统一表示。现有方法通常孤立处理对齐或融合,限制了性能与效率。为此,我们提出新型框架DashFusion。首先,双流对齐模块通过跨模态注意力建立帧级对应关系,同时利用对比学习确保特征空间一致性。其次,监督对比学习引入标签信息优化模态特征。最后,分层瓶颈融合通过压缩瓶颈令牌逐步整合多模态信息,在性能与效率间取得平衡。我们在CMU-MOSI、CMU-MOSEI和CH-SIMS三个数据集上评估,结果表明该方法在多项指标上达到当前最优水平,消融实验验证了对齐与融合机制的有效性。代码已公开于https://github.com/ultramarineX/DashFusion。
原文摘要 · Abstract (English)
Multimodal sentiment analysis (MSA) integrates various modalities, such as text, image, and audio, to provide a more comprehensive understanding of sentiment. However, effective MSA is challenged by alignment and fusion issues. Alignment requires synchronizing both temporal and semantic information across modalities, while fusion involves integrating these aligned features into a unified representation. Existing methods often address alignment or fusion in isolation, leading to limitations in performance and efficiency. To tackle these issues, we propose a novel framework called Dual-stream Alignment with Hierarchical Bottleneck Fusion (DashFusion). Firstly, dual-stream alignment module synchronizes multimodal features through temporal and semantic alignment. Temporal alignment employs cross-modal attention to establish frame-level correspondences among multimodal sequences. Semantic alignment ensures consistency across the feature space through contrastive learning. Secondly, supervised contrastive learning leverages label information to refine the modality features. Finally, hierarchical bottleneck fusion progressively integrates multimodal information through compressed bottleneck tokens, which achieves a balance between performance and computational efficiency. We evaluate DashFusion on three datasets: CMU-MOSI, CMU-MOSEI, and CH-SIMS. Experimental results demonstrate that DashFusion achieves state-of-the-art performance across various metrics, and ablation studies confirm the effectiveness of our alignment and fusion techniques. The codes for our experiments are available at https://github.com/ultramarineX/DashFusion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。