提出DAMM-Diffusion模型,自适应融合多模态肿瘤微环境数据预测纳米颗粒分布。
DAMM-Diffusion: Learning Divergence-Aware Multi-Modal Diffusion Model for Nanoparticles Distribution Prediction
- 设计双分支网络,分别处理单模态与多模态输入,动态选择最优预测路径。
- 引入不确定性感知融合模块,提升跨模态特征对齐精度,实现更高预测准确率。
- 适用于肿瘤微环境分析、医学图像生成等多模态任务,适合医学影像研究者。
纳米颗粒(NPs)分布预测对肿瘤诊断与治疗至关重要。肿瘤微环境(TME)的异质性显著影响NPs在肿瘤中的分布,因此利用多模态TME成分生成分布成为研究热点。然而,多模态数据间的分布差异可能导致联合生成模型性能劣于单一模态模型。为此,本文提出一种分叉感知多模态扩散模型(DAMM-Diffusion),在统一网络中自适应地从单模态与多模态分支生成预测结果。单模态分支采用U-Net架构,多模态分支通过引入两个新模块扩展:多模态融合模块(MMFM)用于多模态特征融合,不确定性感知融合模块(UAFM)用于学习不确定性图以指导交叉注意力计算。基于各分支输出,分叉感知多模态预测器(DAMMP)模块评估多模态数据一致性并决定最终输出来源。实验基于血管与细胞核等TME成分预测纳米颗粒分布,结果表明该模型优于对比方法。额外在多模态脑图像合成任务上也验证了其有效性。
原文摘要 · Abstract (English)
The prediction of nanoparticles (NPs) distribution is crucial for the diagnosis and treatment of tumors. Recent studies indicate that the heterogeneity of tumor microenvironment (TME) highly affects the distribution of NPs across tumors. Hence, it has become a research hotspot to generate the NPs distribution by the aid of multi-modal TME components. However, the distribution divergence among multi-modal TME components may cause side effects i.e., the best uni-modal model may outperform the joint generative model. To address the above issues, we propose a \textbf{D}ivergence-\textbf{A}ware \textbf{M}ulti-\textbf{M}odal \textbf{Diffusion} model (i.e., \textbf{DAMM-Diffusion}) to adaptively generate the prediction results from uni-modal and multi-modal branches in a unified network. In detail, the uni-modal branch is composed of the U-Net architecture while the multi-modal branch extends it by introducing two novel fusion modules i.e., Multi-Modal Fusion Module (MMFM) and Uncertainty-Aware Fusion Module (UAFM). Specifically, the MMFM is proposed to fuse features from multiple modalities, while the UAFM module is introduced to learn the uncertainty map for cross-attention computation. Following the individual prediction results from each branch, the Divergence-Aware Multi-Modal Predictor (DAMMP) module is proposed to assess the consistency of multi-modal data with the uncertainty map, which determines whether the final prediction results come from multi-modal or uni-modal predictions. We predict the NPs distribution given the TME components of tumor vessels and cell nuclei, and the experimental results show that DAMM-Diffusion can generate the distribution of NPs with higher accuracy than the comparing methods. Additional results on the multi-modal brain image synthesis task further validate the effectiveness of the proposed method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。