用分层混合专家模型，跨模态合成缺失的医学影像。
Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Experts
- 分层变分自编码器融合多模态信息，生成高分辨率图像。
- 在脑部多参数MRI与术中超声数据上，合成效果优于现有方法。
- 适合医疗影像补全、多模态融合研究者参考。
我们提出一种深层混合多模态分层变分自编码器（MMHVAE），用于从不同模态的观测图像中合成缺失的图像。该模型针对四个挑战：(i) 构建复杂潜在表示以生成高分辨率图像；(ii) 促使变分分布估计跨模态合成所需的缺失信息；(iii) 在缺失数据背景下学习多模态信息融合；(iv) 利用数据集级信息处理训练时的不完整数据集。在术前脑部多参数磁共振成像与术中超声成像这一具有挑战性的问题上进行了广泛实验。
原文摘要 · Abstract (English)
We propose a deep mixture of multimodal hierarchical variational auto-encoders called MMHVAE that synthesizes missing images from observed images in different modalities. MMHVAE's design focuses on tackling four challenges: (i) creating a complex latent representation of multimodal data to generate high-resolution images; (ii) encouraging the variational distributions to estimate the missing information needed for cross-modal image synthesis; (iii) learning to fuse multimodal information in the context of missing data; (iv) leveraging dataset-level information to handle incomplete data sets at training time. Extensive experiments are performed on the challenging problem of pre-operative brain multi-parametric magnetic resonance and intra-operative ultrasound imaging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。