提出MIDG框架，提升多模态情感分析的跨域泛化能力
MIDG: Mixture of Invariant Experts with knowledge injection for Domain Generalization in Multimodal Sentiment Analysis
- 用混合不变专家模型提取跨模态协同特征
- 通过跨模态适配器注入知识，增强语义表达
- 在3个数据集上表现优于现有方法
多模态情感分析（MSA）中的领域泛化方法常忽视模态间的协同作用，导致难以准确捕捉多模态数据中的丰富语义信息。尽管已有知识注入技术应用于MSA，但通常存在跨模态知识碎片化问题，忽略超越单模态的特定表示。为此，我们提出一种新的MSA领域泛化框架。首先，引入混合不变专家模型以提取领域不变特征，增强模态间协同关系的学习能力。其次，设计跨模态适配器，通过跨模态知识注入提升多模态表征的语义丰富度。在三个数据集上的广泛实验表明，所提MIDG框架性能显著更优。
原文摘要 · Abstract (English)
Existing methods in domain generalization for Multimodal Sentiment Analysis (MSA) often overlook inter-modal synergies during invariant features extraction, which prevents the accurate capture of the rich semantic information within multimodal data. Additionally, while knowledge injection techniques have been explored in MSA, they often suffer from fragmented cross-modal knowledge, overlooking specific representations that exist beyond the confines of unimodal. To address these limitations, we propose a novel MSA framework designed for domain generalization. Firstly, the framework incorporates a Mixture of Invariant Experts model to extract domain-invariant features, thereby enhancing the model's capacity to learn synergistic relationships between modalities. Secondly, we design a Cross-Modal Adapter to augment the semantic richness of multimodal representations through cross-modal knowledge injection. Extensive domain experiments conducted on three datasets demonstrate that the proposed MIDG achieves superior performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。