用强化学习动态选模态,提升肺癌风险预测准确率
AdaFuse: Adaptive Multimodal Fusion for Lung Cancer Risk Prediction via Reinforcement Learning
- 通过强化学习逐轮决定是否引入新模态,实现个性化融合
- 在NLST数据集上达AUC 0.762,优于单模态和固定融合方法
- 能提前终止冗余模态,节省计算量,适合临床高效诊断
多模态融合已成为疾病诊断与预后的有力范式,整合医学影像、临床记录和放射科报告等异构数据。然而现有方法对所有可用模态进行统一处理,或学习分配不同权重,未解决关键问题:对特定患者,是否应使用某些模态?本文提出AdaFuse,一种基于强化学习的自适应多模态融合框架,用于肺癌风险预测。该框架将多模态融合建模为序列决策过程,策略网络迭代判断是否引入额外模态或进入预测阶段,依据已获取信息动态调整。此机制使每次选择依赖先前模态,并在信息足够时提前终止,而非预先确定固定子集。在国家肺筛查试验(NLST)数据集上评估显示,AdaFuse达到最高AUC 0.762,优于最佳单模态基线(0.732)、最佳固定融合策略(0.759),以及自适应基线如DynMM(0.754)和MoE(0.742),且比所有三模态方法更少使用浮点运算量(FLOPs)。本工作展示了强化学习在医学影像个性化多模态融合中的潜力,推动诊断流程从统一融合向自适应策略转变。
原文摘要 · Abstract (English)
Multimodal fusion has emerged as a promising paradigm for disease diagnosis and prognosis, integrating complementary information from heterogeneous data sources such as medical images, clinical records, and radiology reports. However, existing fusion methods process all available modalities through the network, either treating them equally or learning to assign different contribution weights, leaving a fundamental question unaddressed: for a given patient, should certain modalities be used at all? We present AdaFuse, an adaptive multimodal fusion framework that leverages reinforcement learning (RL) to learn patient-specific modality selection and fusion strategies for lung cancer risk prediction. AdaFuse formulates multimodal fusion as a sequential decision process, where the policy network iteratively decides whether to incorporate an additional modality or proceed to prediction based on the information already acquired. This sequential formulation enables the model to condition each selection on previously observed modalities and terminate early when sufficient information is available, rather than committing to a fixed subset upfront. We evaluate AdaFuse on the National Lung Screening Trial (NLST) dataset. Experimental results demonstrate that AdaFuse achieves the highest AUC (0.762) compared to the best single-modality baseline (0.732), the best fixed fusion strategy (0.759), and adaptive baselines including DynMM (0.754) and MoE (0.742), while using fewer FLOPs than all triple-modality methods. Our work demonstrates the potential of reinforcement learning for personalized multimodal fusion in medical imaging, representing a shift from uniform fusion strategies toward adaptive diagnostic pipelines that learn when to consult additional modalities and when existing information suffices for accurate prediction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。