提出多模态不确定性传播模型,可量化医学影像与文本的联合不确定性。
Analysis of Image-and-Text Uncertainty Propagation in Multimodal Large Language Models with Cardiac MR-Based Applications
- 基于不确定性传播构建跨模态分析框架
- 仅需少量样本即可稳健优化模型,且在不同任务间具有强泛化性
- 适用于心脏疾病预测等临床场景,支持不确定性的可解释分析
多模态大语言模型(MLLMs)可处理图像与文本等多源信息,但其输入模态间的相互关系、单模态数据带来的不确定性,以及不确定性分解后的临床应用仍不明确。本文提出多模态不确定性传播模型(MUPM),通过不确定性传播机制刻画图像、文本及联合输入所引发的不确定性关系。利用真实心脏磁共振影像与数字健康记录数据,验证了MUPM可在少量样本下稳健优化,并具备跨不同数据分布与下游任务的泛化能力。该可迁移性可能源于共同预训练、轻量微调及低维模型结构。更重要的是,该模型能有效估计并分析不同数据下的不确定性,直接支持心脏疾病预测等新任务的鲁棒性评估。实验还表明,该方法在少样本条件下高效估算整体不确定性,同时识别冗余因素,具有临床实用价值。代码已开源:https://github.com/yucheng722/MUPM。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) can process and integrate information from multimodality sources, such as text and images. However, interrelationship among input modalities, uncertainties due to individual uni-modal data and potential clinical applications following such an uncertainty decomposition are yet fully understood in the context of large-scale MLLMs. In this work, we propose a multimodal uncertainty propagation model (MUPM) based on uncertainty propagation, to characterise the relationship among the uncertainties arising from image-only, text-only, and joint image-text variations in MLLM inputs. Using real clinical data consisting of cardiac MR scans and digital health records, we describe that MUPMs can be optimised robustly with a few samples. We then show that the fitted MUPMs are generalisable across different input data distributions and, perhaps surprisingly, across different downstream tasks. Such a transferability may be explained by the shared pretraining, comparatively light MLLM fine-tuning, along with the low-dimensional nature of the MUPMs. More importantly, this learned transferability, quantifying the relationship between these uncertainties, led to direct clinical applications in which uncertainties may be estimated and thus analysed robustly for varying data or even a novel set of cardiac disease prediction tasks. In addition, we show experimentally the efficiency in multimodal data required for estimating the overall uncertainty and its ability to identify redundant factors, both of which are considered practical yet clinically useful applications with the proposed MUPMs. Codes are available at https://github.com/yucheng722/MUPM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。