针对多模态情感分析数据少、噪声高的问题,提出自适应情感增强方法。
MS-Mix: Sentiment-Guided Adaptive Augmentation for Multimodal Sentiment Analysis
- 按情绪语义相似性筛选样本,避免跨模态情绪冲突。
- 根据情感强度动态调整各模态混合比例,提升一致性。
- 引入跨模态情感分布对齐损失,显著增强模型鲁棒性。
多模态情感分析(MSA)融合文本、视频和音频的互补特征,以实现对人类交互中情感的稳健理解。然而,现有模型受限于严重的数据稀缺性和高昂的标注成本,极大制约了其在社交媒体分析与人机系统中的实际应用。现有的基于Mixup的增强技术若直接应用于MSA,常因忽略模态间的情感语义而生成语义不一致的样本,并放大标签噪声。为此,本文提出MS-Mix,一种自适应的情绪敏感增强框架,可自动优化多模态数据质量。其核心组件包括:(1) 情绪感知的样本选择策略,通过潜在空间语义相似性过滤不兼容样本,防止情绪矛盾混合;(2) 基于多头自注意力的情感强度引导模块,动态计算各模态的混合比例,以适应情感显著性;(3) 基于KL散度的情感对齐损失,使各模态预测的情感分布与真实标签对齐,提升判别力与一致性。在两个公开数据集上,使用六种主流骨干网络的大量实验表明,MS-Mix持续优于现有方法,显著提升了多模态情感分析的鲁棒性与实用性。源代码见匿名链接:https://anonymous.4open.science/r/MS-Mix-review-0C72。
原文摘要 · Abstract (English)
Multimodal Sentiment Analysis (MSA) integrates complementary features from text, video, and audio for robust emotion understanding in human interactions. However, models suffer from severe data scarcity and high annotation costs, severely limiting real-world deployment in social media analytics and human-computer systems. Existing Mixup-based augmentation techniques, when naively applied to MSA, often produce semantically inconsistent samples and amplified label noise by ignoring emotional semantics across modalities. To address these challenges, we propose MS-Mix, an adaptive emotion-sensitive augmentation framework that automatically optimizes data quality in multimodal settings. Its key components are: (1) Sentiment-aware sample selection strategy that filters incompatible pairs via latent-space semantic similarity to prevent contradictory emotion mixing. (2) Sentiment intensity guided module with multi-head self-attention for computing modality-specific mixing ratios conditioned on emotional salience dynamically. (3) Sentiment alignment loss based on Kullback-Leibler divergence to align predicted sentiment distributions across modalities with ground-truth labels, improving discrimination and consistency. Extensive experiments on two public datasets with six state-of-the-art backbones confirm that MS-Mix consistently outperforms prior methods, significantly improving robustness and practical applicability for MSA. The source code is available at an anonymous link: https://anonymous.4open.science/r/MS-Mix-review-0C72.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。