SimMLM让多模态模型在缺失数据时仍能稳定准确，无需复杂设计。
SimMLM: A Simple Framework for Multi-modal Learning with Missing Modality
- 用动态门控机制自动调节各模态贡献，适应完整或缺失场景。
- 引入新损失函数，确保多模态加入不降低准确率，反而提升。
- 在医学图像分割与分类任务中表现更优，适合真实缺失数据场景。
本文提出 SimMLM，一种简单而强大的多模态学习框架，适用于模态缺失情况。不同于依赖复杂网络结构或数据补全的方法，SimMLM 采用通用的动态模态专家混合（DMoME）架构，通过可学习的动态门控机制，在完整和部分模态设置下自动调整各模态贡献。其核心创新是提出的“更多优于更少”（MoFe）排序损失，确保随着可用模态增多，任务准确率不降反升，符合直观原则：删减模态不应提高性能。在 BraTS 2018 多模态医学图像分割和 UPMC Food-101、avMNIST 多模态分类任务上，该方法持续超越对比方法，展现出更高的准确性、可解释性、鲁棒性和可靠性，适用于测试时存在模态缺失的实际场景。
原文摘要 · Abstract (English)
In this paper, we propose SimMLM, a simple yet powerful framework for multimodal learning with missing modalities. Unlike existing approaches that rely on sophisticated network architectures or complex data imputation techniques, SimMLM provides a generic and effective solution that can adapt to various missing modality scenarios with improved accuracy and robustness. Specifically, SimMLM consists of a generic Dynamic Mixture of Modality Experts (DMoME) architecture, featuring a dynamic, learnable gating mechanism that automatically adjusts each modality's contribution in both full and partial modality settings. A key innovation of SimMLM is the proposed More vs. Fewer (MoFe) ranking loss, which ensures that task accuracy improves or remains stable as more modalities are made available. This aligns the model with an intuitive principle: removing one or more modalities should not increase accuracy. We validate SimMLM on multimodal medical image segmentation (BraTS 2018) and multimodal classification (UPMC Food-101, avMNIST) tasks, where it consistently surpasses competitive methods, demonstrating superior accuracy, interpretability, robustness, and reliability across both complete and missing modality scenarios at test time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。