arXiv:2504.12151cs.LGcs.AI2025-04被引 12

提升多模态情感分析的可解释性与模态平衡,兼顾性能与透明度。

Towards Explainable Fusion and Balanced Learning in Multimodal Sentiment Analysis

  • 用KAN网络分解跨模态交互,实现决策过程的直接可视化。
  • 通过降维去噪瓶颈机制,降低模态不平衡并保留关键情感信息。
  • 在CMU-MOSI等数据集上表现优异,适合需要可信AI的场景。

多模态情感分析面临两大挑战:融合决策缺乏可解释性,以及因模态间信息密度差异导致的模态不平衡。为此,我们提出KAN-MCP框架,结合柯尔莫哥洛夫-阿诺德网络(KAN)的可解释性与多模态纯净帕累托(MCPareto)框架的鲁棒性。首先,KAN利用其单变量函数分解特性,实现跨模态交互的透明分析,无需外部解释工具即可直观观察特征变换过程,兼具高表达力与可解释性。其次,提出的MCPareto通过引入维度缩减与去噪模态信息瓶颈(DRD-MIB)方法,联合实现降维与去噪,为KAN提供判别性强的低维输入,降低建模复杂度同时保留关键情感信息。此外,基于DRD-MIB输出的净化特征,动态调节各模态梯度贡献,实现辅助信号无损传递,有效缓解模态不平衡问题。该协同设计在CMU-MOSI、CMU-MOSEI和CH-SIMS v2等基准数据集上取得优越性能,并通过KAN的可解释架构提供直观可视化界面。代码已开源:https://github.com/LuoMSen/KAN-MCP。

原文摘要 · Abstract (English)

Multimodal Sentiment Analysis (MSA) faces two critical challenges: the lack of interpretability in the decision logic of multimodal fusion and modality imbalance caused by disparities in inter-modal information density. To address these issues, we propose KAN-MCP, a novel framework that integrates the interpretability of Kolmogorov-Arnold Networks (KAN) with the robustness of the Multimodal Clean Pareto (MCPareto) framework. First, KAN leverages its univariate function decomposition to achieve transparent analysis of cross-modal interactions. This structural design allows direct inspection of feature transformations without relying on external interpretation tools, thereby ensuring both high expressiveness and interpretability. Second, the proposed MCPareto enhances robustness by addressing modality imbalance and noise interference. Specifically, we introduce the Dimensionality Reduction and Denoising Modal Information Bottleneck (DRD-MIB) method, which jointly denoises and reduces feature dimensionality. This approach provides KAN with discriminative low-dimensional inputs to reduce the modeling complexity of KAN while preserving critical sentiment-related information. Furthermore, MCPareto dynamically balances gradient contributions across modalities using the purified features output by DRD-MIB, ensuring lossless transmission of auxiliary signals and effectively alleviating modality imbalance. This synergy of interpretability and robustness not only achieves superior performance on benchmark datasets such as CMU-MOSI, CMU-MOSEI, and CH-SIMS v2 but also offers an intuitive visualization interface through KAN's interpretable architecture. Our code is released on https://github.com/LuoMSen/KAN-MCP.

多模态情感分析可解释性模态平衡

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。