让多模态融合更智能：自动识别并解释不同模态间的交互关系。
I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts
- 用多个专家网络学习模态间复杂交互，弱监督驱动数据驱动建模。
- 通过重加权机制提升性能，在医疗和通用数据集上均显著优于基线。
- 既可提升模型效果，又能解释每条样本的模态贡献，适合需要可解释性的场景。
模态融合是多模态学习的核心，能整合来自不同数据源的信息。然而，传统融合方法存在两大局限：（1）无法捕捉模态间的异质性交互；（2）缺乏对数据中内在多模态交互的可解释性。为此，我们提出 I2MoE（Interpretable Multimodal Interaction-aware Mixture of Experts），一个端到端的 MoE 框架，旨在通过显式建模多样化的多模态交互来增强模态融合，并提供局部与全局层面的可解释性。首先，I2MoE 利用多个具有弱监督交互损失的交互专家，以数据驱动方式学习多模态交互。其次，部署重加权模型为每个交互专家的输出分配重要性评分，实现样本级与数据集级的解释。在医学和通用多模态数据集上的大量实验表明，I2MoE 足够灵活，可与多种融合技术结合，持续提升任务性能，并在多种真实场景中提供可解释性。代码已开源：https://github.com/Raina-Xin/I2MoE。
原文摘要 · Abstract (English)
Modality fusion is a cornerstone of multimodal learning, enabling information integration from diverse data sources. However, vanilla fusion methods are limited by (1) inability to account for heterogeneous interactions between modalities and (2) lack of interpretability in uncovering the multimodal interactions inherent in the data. To this end, we propose I2MoE (Interpretable Multimodal Interaction-aware Mixture of Experts), an end-to-end MoE framework designed to enhance modality fusion by explicitly modeling diverse multimodal interactions, as well as providing interpretation on a local and global level. First, I2MoE utilizes different interaction experts with weakly supervised interaction losses to learn multimodal interactions in a data-driven way. Second, I2MoE deploys a reweighting model that assigns importance scores for the output of each interaction expert, which offers sample-level and dataset-level interpretation. Extensive evaluation of medical and general multimodal datasets shows that I2MoE is flexible enough to be combined with different fusion techniques, consistently improves task performance, and provides interpretation across various real-world scenarios. Code is available at https://github.com/Raina-Xin/I2MoE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。