动态调整模态权重，让模型自动选主导信息提升情感分析效果
Knowledge-Guided Dynamic Modality Attention Fusion Framework for Multimodal Sentiment Analysis
- 用情感知识引导模型动态选择主导模态
- 在4个数据集上达到当前最佳性能
- 适合需要灵活处理多模态输入的场景
多模态情感分析利用多源数据推断用户情感。以往方法通常平等地对待各模态贡献，或静态地以文本为主导进行交互，忽略了不同情境下各模态可能成为主导的情况。本文提出知识引导的动态模态注意力融合框架（KuDA），利用情感知识动态指导模型选择主导模态并调整各模态贡献。此外，在获得多模态表示后，模型通过相关性评估损失进一步强化主导模态的贡献。在四个MSA基准数据集上的大量实验表明，KuDA取得了当前最优性能，并能适应不同主导模态的情境。
原文摘要 · Abstract (English)
Multimodal Sentiment Analysis (MSA) utilizes multimodal data to infer the users' sentiment. Previous methods focus on equally treating the contribution of each modality or statically using text as the dominant modality to conduct interaction, which neglects the situation where each modality may become dominant. In this paper, we propose a Knowledge-Guided Dynamic Modality Attention Fusion Framework (KuDA) for multimodal sentiment analysis. KuDA uses sentiment knowledge to guide the model dynamically selecting the dominant modality and adjusting the contributions of each modality. In addition, with the obtained multimodal representation, the model can further highlight the contribution of dominant modality through the correlation evaluation loss. Extensive experiments on four MSA benchmark datasets indicate that KuDA achieves state-of-the-art performance and is able to adapt to different scenarios of dominant modality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。