用扩散Transformer同时生成小鼠眼OCT图像和分割掩码,解决标注数据少的问题。
DualDiT: A Conditional Dual-Output Diffusion Transformer for Joint OCT Image and Segmentation Mask Generation

- 设计双输出扩散Transformer,联合建模图像与分割掩码的分布。
- 生成图像FID 56.14、sFID 114.35,专家误判率超四成。
- 生成数据可提升下游分割模型性能,适合医学图像数据增强场景。
生成真实且解剖结构准确的医学图像及其分割掩码,有助于缓解医学影像中标注数据不足的问题,尤其在小鼠眼光学相干断层扫描(OCT)中,由于结构微小且需专业技能,手动划分视网膜层耗时费力,导致数据稀缺。尽管扩散模型在医学图像合成中表现优异,但联合图像-掩码生成仍主要依赖基于U-Net的去噪器,扩散变压器尚未被充分探索。本文提出条件双输出扩散变压器(DualDiT),用于联合生成离体小鼠视网膜上部视网膜细胞层的OCT B-scan图像与分割掩码。DualDiT通过预训练VAE将两种模态编码至共享潜在空间,拼接其潜在表示,并在联合张量上进行条件扩散。与两种改编的扩散基线(DDPM和LDM)对比,评估指标包括弗雷切特图像距离(FID)和空间FID(sFID)、下游U-Net分割任务中的数据增强效果,以及三位领域专家的感知真实性评估。结果表明,DualDiT在生成质量上最优(FID 56.14,sFID 114.35),优于基线;专家小组将46%的合成样本误判为真实,42%的真实样本误判为合成。加入由DualDiT生成的图像和掩码后,持留测试集上的Dice和IoU得分均提升。结论:变压器驱动的扩散模型可有效学习OCT图像与分割掩码的联合分布,其生成保真度、下游实用性及感知真实性均优于基于DDPM和LDM的基线,凸显其在标注稀缺医学影像中数据增强的潜力。
原文摘要 · Abstract (English)
Background and Objective: Generating realistic medical images with anatomically accurate segmentation masks helps address the shortage of annotated data in medical imaging, particularly in optical coherence tomography (OCT) of mouse eyes, where manual retinal layer delineation is labour-intensive due to tiny structures and required expertise, resulting in scarce datasets. While diffusion models perform well in medical image synthesis, joint image-mask generation has relied mainly on U-Net-based denoisers, leaving diffusion transformers largely unexplored. Methods: We propose a conditional dual-output Diffusion Transformer (DualDiT) for joint synthesis of OCT B-scans and segmentation masks of the upper retinal cell layers in ex vivo mouse retina. DualDiT encodes both modalities into a shared latent space via a pretrained VAE, concatenates their latent representations, and performs conditional diffusion over the joint tensor. We compared DualDiT against two adapted diffusion baselines: DDPM and LDM. Generative quality was assessed via Fréchet Inception Distance (FID) and spatial FID (sFID); practical utility via synthetic data augmentation for downstream U-Net segmentation; and perceptual realism via evaluation by three domain experts. Results: DualDiT achieved the best generative quality (FID 56.14, sFID 114.35), outperforming DDPM and LDM. Expert panels misclassified 46% of synthetic samples as real and 42% of real samples as synthetic. Adding DualDiT-generated images and masks improved Dice and IoU scores on a held-out segmentation test set. Conclusions: DualDiT shows that transformer-based diffusion models can effectively learn the joint distribution of OCT images and segmentation masks, surpassing DDPM- and LDM-based baselines in generative fidelity, downstream utility, and perceptual realism, highlighting its potential for data augmentation in annotation-scarce medical imaging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。