用Mamba模型融合影像与表格数据,提升癌症诊断可解释性。
An approach with Visual and Tabular Mamba to multimodal medical data using Mixed Fusion

- 分路处理影像与临床数据,通过混合融合架构整合多模态信息。
- 在口腔癌数据集上表现优于Transformer,召回率显著提升。
- 支持SHAP解释方法,适合对诊断可解释性要求高的医疗场景。
本文提出一种基于状态空间模型Mamba的多模态医学数据融合方法,用于癌症分类。设计了混合融合架构(Mixed Fusion),分别用Mamba处理视觉数据(皮肤病变图像)和表格数据(临床/社会人口学信息),前者生成类别概率,后者结合概率与临床数据输出最终诊断。在PAD-UFES-20(皮肤病变)和NDB-UFES(口腔癌)两个数据集上测试,结果表明:在PAD-UFES-20上平衡准确率略低于Transformer,在NDB-UFES上表现更优,且召回率有显著提升。该架构支持使用SHAP方法进行解释,增强了决策过程的可解释性。研究显示,基于Mamba的模型在敏感性要求高的医疗场景中具有应用潜力。
原文摘要 · Abstract (English)
This article presents a complementary approach for integrating multimodal medical data in cancer classification, based on state space models represented by the Mamba architecture. To this end, a mixed multimodal fusion architecture, called Mixed Fusion, was employed and developed to enhance the interpretability of the decision-making process. The proposed approach explores two variants of Mamba: one dedicated to visual processing, responsible for classifying the lesion image and generating probabilities associated with the target classes, and another focused on tabular processing, which uses these probabilities together with clinical and/or sociodemographic data to produce the final diagnosis. The experiments were conducted on two medical datasets: PAD-UFES-20, composed of clinical images and information associated with skin lesions, and NDB-UFES, consisting of histopathological images and sociodemographic data related to oral cancer. The results indicate slightly lower performance in balanced accuracy, compared with Transformer-based approaches, on PAD-UFES-20, and superior performance on NDB-UFES. Additionally, substantial gains were observed in the recall metric. Furthermore, the adoption of the Mixed Fusion architecture enables the application of the Shapley Additive Explanations (SHAP) method, increasing the interpretability of the results. These findings indicate that Mamba-based models constitute a suitable alternative for multimodal classification in medical data, especially in scenarios in which sensitivity is a relevant requirement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。