arXiv:2412.12225cs.LGcs.AI2024-12AAAI被引 126

分离语言与多模态特征,让语言主导情感分析。

DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis

论文配图:DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis
图 1 · 摘自论文原文
  • 拆分共享与特定模态信息,减少冗余干扰
  • 语言引导注意力机制提升文本特征表达
  • 适合关注语言在情感判断中作用的研究者

多模态情感分析(MSA)利用语言、视觉和音频等异构模态提升对人类情感的理解。现有模型通常聚焦于跨模态共享信息的提取或直接融合不同模态,但这种对所有模态一视同仁的方法会引入冗余和模态间冲突。为此,我们提出一种解耦-语言聚焦(DLF)多模态表征学习框架,包含特征解耦模块,用于分离模态共享与模态特定信息;引入四种几何度量优化解耦过程,进一步降低冗余并增强语言相关特征。此外,设计语言聚焦吸引器(LFA),通过语言引导的交叉注意力机制,利用其他模态的特定信息强化语言表示。框架还采用分层预测策略提升整体准确率。在两个主流数据集CMU-MOSI和CMU-MOSEI上的大量实验表明,所提DLF框架显著提升了性能。消融实验证实了特征解耦模块、语言聚焦吸引器及分层预测的有效性。代码已开源:https://github.com/pwang322/DLF。

原文摘要 · Abstract (English)

Multimodal Sentiment Analysis (MSA) leverages heterogeneous modalities, such as language, vision, and audio, to enhance the understanding of human sentiment. While existing models often focus on extracting shared information across modalities or directly fusing heterogeneous modalities, such approaches can introduce redundancy and conflicts due to equal treatment of all modalities and the mutual transfer of information between modality pairs. To address these issues, we propose a Disentangled-Language-Focused (DLF) multimodal representation learning framework, which incorporates a feature disentanglement module to separate modality-shared and modality-specific information. To further reduce redundancy and enhance language-targeted features, four geometric measures are introduced to refine the disentanglement process. A Language-Focused Attractor (LFA) is further developed to strengthen language representation by leveraging complementary modality-specific information through a language-guided cross-attention mechanism. The framework also employs hierarchical predictions to improve overall accuracy. Extensive experiments on two popular MSA datasets, CMU-MOSI and CMU-MOSEI, demonstrate the significant performance gains achieved by the proposed DLF framework. Comprehensive ablation studies further validate the effectiveness of the feature disentanglement module, language-focused attractor, and hierarchical predictions. Our code is available at https://github.com/pwang322/DLF.

多模态情感分析解耦学习语言聚焦

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。