自动找多模态医学影像最佳融合时机,提升诊断准确率。
Timing Is Everything: Finding the Optimal Fusion Points in Multimodal Medical Imaging
- 逐层测试融合模块,用验证损失选最优位置。
- 在两个MRI数据集上均超越单模态和全组合方法。
- 计算开销小,适合临床级医疗AI系统部署。
多模态深度学习通过整合如MRI序列等不同成像模态,提升医学影像诊断准确性。关键挑战在于确定模态融合的最佳时机——即融合模块应插入网络的哪一层。现有方法依赖人工调参或穷举搜索,计算成本高且无法保证找到最优解。本文提出一种顺序前向搜索算法,逐步激活并评估不同网络层的候选融合模块。每一步基于先前学习的权重重新训练,并比较验证损失以确定最优配置。该过程系统性缩小搜索空间,实现高效定位最优融合时机,无需穷尽所有可能的模块放置。在两个针对不同分类任务的多模态MRI数据集上验证,所获架构持续优于单模态基线、晚期融合及全组合式集成方案。其在准确率、F-score和特异性方面表现更优,同时保持竞争性或更高的AUC值。此外,搜索过程的顺序特性显著降低计算开销,使优化更具实用性。本方法系统性地确定了模态融合的最优时机,推动了医学影像多模态深度学习的发展,为临床决策支持提供高效可靠的融合优化框架,助力可扩展、自适应的医疗AI架构演进。
原文摘要 · Abstract (English)
Multimodal deep learning harnesses diverse imaging modalities, such as MRI sequences, to enhance diagnostic accuracy in medical imaging. A key challenge is determining the optimal timing for integrating these modalities-specifically, identifying the network layers where fusion modules should be inserted. Current approaches often rely on manual tuning or exhaustive search, which are computationally expensive without any guarantee of converging to optimal results. We propose a sequential forward search algorithm that incrementally activates and evaluates candidate fusion modules at different layers of a multimodal network. At each step, the algorithm retrains from previously learned weights and compares validation loss to identify the best-performing configuration. This process systematically reduces the search space, enabling efficient identification of the optimal fusion timing without exhaustively testing all possible module placements. The approach is validated on two multimodal MRI datasets, each addressing different classification tasks. Our algorithm consistently identified configurations that outperformed unimodal baselines, late fusion, and a brute-force ensemble of all potential fusion placements. These architectures demonstrated superior accuracy, F-score, and specificity while maintaining competitive or improved AUC values. Furthermore, the sequential nature of the search significantly reduced computational overhead, making the optimization process more practical. By systematically determining the optimal timing to fuse imaging modalities, our method advances multimodal deep learning for medical imaging. It provides an efficient and robust framework for fusion optimization, paving the way for improved clinical decision-making and more adaptable, scalable architectures in medical AI applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。