用扩散模型提升多模态医学图像融合质量,细节更清晰、信息更丰富。
DM-FNet: Unified multimodal medical image fusion via diffusion process-trained encoder-decoder
- 分两阶段训练:先用扩散模型重建图像,再融合多模态特征
- 在多种医学图像上优于现有方法,亮度、对比度和纹理均更优
- 适合需要高精度图像融合的临床诊断与医学影像研究者
多模态医学图像融合(MMIF)从多源图像中提取关键信息,提升诊断全面性与准确性。高质量融合需平衡亮度、色彩、对比度与细节,以清晰呈现解剖结构并反映组织功能状态。现有方法在特征捕获与跨模态交互方面能力有限,导致融合效果不佳。本文提出基于两阶段扩散模型的融合网络(DM-FNet)。第一阶段,通过扩散过程训练UNet进行图像重建,利用逐步去噪捕捉多层级细节特征;第二阶段,将不同噪声步长的图像输入融合网络,增强特征识别能力,并引入三个自适应融合模块处理多模态数据。最终结合稳健网络结构与混合损失函数,统一优化亮度、色彩、对比度与细节。实验表明,该方法在多种医学图像上表现优异,融合图像保持合理亮度、放射性示踪剂分布全面、纹理丰富、边缘清晰。代码已开源:https://github.com/HeDan-11/DM-FNet。
原文摘要 · Abstract (English)
Multimodal medical image fusion (MMIF) extracts the most meaningful information from multiple source images, enabling a more comprehensive and accurate diagnosis. Achieving high-quality fusion results requires a careful balance of brightness, color, contrast, and detail; this ensures that the fused images effectively display relevant anatomical structures and reflect the functional status of the tissues. However, existing MMIF methods have limited capacity to capture detailed features during conventional training and suffer from insufficient cross-modal feature interaction, leading to suboptimal fused image quality. To address these issues, this study proposes a two-stage diffusion model-based fusion network (DM-FNet) to achieve unified MMIF. In Stage I, a diffusion process trains UNet for image reconstruction. UNet captures detailed information through progressive denoising and represents multilevel data, providing a rich set of feature representations for the subsequent fusion network. In Stage II, noisy images at various steps are input into the fusion network to enhance the model's feature recognition capability. Three key fusion modules are also integrated to process medical images from different modalities adaptively. Ultimately, the robust network structure and a hybrid loss function are integrated to harmonize the fused image's brightness, color, contrast, and detail, enhancing its quality and information density. The experimental results across various medical image types demonstrate that the proposed method performs exceptionally well regarding objective evaluation metrics. The fused image preserves appropriate brightness, a comprehensive distribution of radioactive tracers, rich textures, and clear edges. The code is available at https://github.com/HeDan-11/DM-FNet.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。