对比多种医学多模态融合方法,揭示何时能提升诊断准确率。
When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion
- 设计可插拔的基准工具,测试电子病历与胸片融合效果
- 模态完整时性能提升,但缺失数据下效果迅速下降
- 模型复杂度无法克服电子病历的时间结构偏差
机器学习有望推动临床决策支持,但其在真实场景中何时真正有效仍不明确,尤其在模态缺失和公平性约束下。本文基于MIMIC-IV和MIMIC-CXR标准数据集,系统评估电子健康记录(EHR)与胸部X光(CXR)的多模态融合,回答四个核心问题:何时融合提升预测性能、不同融合策略表现如何、现有方法对缺失模态的鲁棒性如何、多模态模型是否实现算法公平。研究发现:当模态完整时,融合可提升性能,增益集中于需双模态互补信息的疾病;跨模态学习机制捕捉到超越简单拼接的临床关联,但EHR的丰富时间结构导致显著模态不平衡,仅靠架构复杂度无法解决;在现实缺失情况下,除非模型显式处理不完整输入,否则多模态优势快速消失;多模态融合不自带公平性,子群体差异主要源于不同人群敏感性不均。为支持可复现评估,我们开源了灵活的基准工具包,支持新模型与数据集即插即用。该工作为构建高效可靠的临床部署多模态系统提供实证指导。
原文摘要 · Abstract (English)
Machine learning holds promise for advancing clinical decision support, yet it remains unclear when multimodal learning truly helps in practice, particularly under modality missingness and fairness constraints. In this work, we conduct a systematic benchmark of multimodal fusion between Electronic Health Records (EHR) and chest X-rays (CXR) on standardized cohorts from MIMIC-IV and MIMIC-CXR, aiming to answer four fundamental questions: when multimodal fusion improves clinical prediction, how different fusion strategies compare, how robust existing methods are to missing modalities, and whether multimodal models achieve algorithmic fairness. Our study reveals several key insights. Multimodal fusion improves performance when modalities are complete, with gains concentrating in diseases that require complementary information from both EHR and CXR. While cross-modal learning mechanisms capture clinically meaningful dependencies beyond simple concatenation, the rich temporal structure of EHR introduces strong modality imbalance that architectural complexity alone cannot overcome. Under realistic missingness, multimodal benefits rapidly degrade unless models are explicitly designed to handle incomplete inputs. Moreover, multimodal fusion does not inherently improve fairness, with subgroup disparities mainly arising from unequal sensitivity across demographic groups. To support reproducible and extensible evaluation, we further release a flexible benchmarking toolkit that enables plug-and-play integration of new models and datasets. Together, this work provides actionable guidance on when multimodal learning helps, when it fails, and why, laying the foundation for developing clinically deployable multimodal systems that are both effective and reliable. The open-source toolkit can be found at https://github.com/jakeykj/CareBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。