实现医学影像多模态灵活转换,支持任意K到N的生成。
Med-K2N: Flexible K-to-N Modality Translation for Medical Image Synthesis
- 将多模态影像视为有序序列,动态学习各模态权重。
- 在多个数据集上超越现有方法,显著提升生成质量。
- 适合需要灵活生成缺失医学影像的临床研究者。
跨模态医学图像合成旨在从已有模态重建缺失影像以辅助临床诊断。针对临床中对灵活模态重建的需求,本文探索了K to N医学图像生成任务,提出三大挑战:如何建模不同模态对各类目标任务的异质贡献?如何确保融合质量以避免噪声信息导致性能下降?如何在多输出生成中保持模态身份一致性?受临床医生逐步添加与选择性整合多模态信息的工作流程启发,我们借鉴SAM2的逐帧处理范式,将多模态医学数据视为带质量驱动选择机制的序列帧。核心思想是为每对模态-任务学习自适应权重,并通过渐进增强“记忆”有益融合模式。为此设计三个协同模块:PreWeightNet用于全局贡献评估,ThresholdNet实现自适应过滤,EffiWeightNet计算有效权重。为保持模态身份一致性,提出因果模态身份模块(CMIM),利用视觉-语言建模建立生成图像与目标模态描述间的因果约束。大量实验表明,Med-K2N在多个基准测试中显著优于现有先进方法。源代码已公开。
原文摘要 · Abstract (English)
Cross-modal medical image synthesis research focuses on reconstructing missing imaging modalities from available ones to support clinical diagnosis. Driven by clinical necessities for flexible modality reconstruction, we explore K to N medical generation, where three critical challenges emerge: How can we model the heterogeneous contributions of different modalities to various target tasks? How can we ensure fusion quality control to prevent degradation from noisy information? How can we maintain modality identity consistency in multi-output generation? Driven by these clinical necessities, and drawing inspiration from SAM2's sequential frame paradigm and clinicians' progressive workflow of incrementally adding and selectively integrating multi-modal information, we treat multi-modal medical data as sequential frames with quality-driven selection mechanisms. Our key idea is to "learn" adaptive weights for each modality-task pair and "memorize" beneficial fusion patterns through progressive enhancement. To achieve this, we design three collaborative modules: PreWeightNet for global contribution assessment, ThresholdNet for adaptive filtering, and EffiWeightNet for effective weight computation. Meanwhile, to maintain modality identity consistency, we propose the Causal Modality Identity Module (CMIM) that establishes causal constraints between generated images and target modality descriptions using vision-language modeling. Extensive experimental results demonstrate that our proposed Med-K2N outperforms state-of-the-art methods by significant margins on multiple benchmarks. Source code is available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。