arXiv:2603.25181cs.CV2026-03

用纯Transformer生成3D医学影像,更可控、更连贯。

VolDiT: Controllable Volumetric Medical Image Synthesis with Diffusion Transformers

  • 用3D分块嵌入和全局自注意力实现纯Transformer的体积图像生成。
  • 相比传统U-Net模型,生成图像全局一致性更强,细节更真实。
  • 支持分割图控制生成,适合需要精准空间引导的研究者。

扩散模型已成为高保真医学图像合成的主流方法。然而,现有3D医学图像生成大多基于潜空间扩散框架中的卷积U-Net主干网络。这类架构具有强局部性偏差和有限感受野,限制了可扩展性、全局上下文整合及灵活条件控制。本文提出VolDiT,首个专用于体积医学图像合成的纯Transformer型3D扩散模型。通过体积分块嵌入与直接作用于3D标记的全局自注意力,将扩散变压器扩展至原生3D数据。为实现结构化控制,提出时间步门控控制适配器,将分割掩码映射为可学习控制标记,在去噪过程中调制Transformer层。该标记级条件机制在保持Transformer建模优势的同时,实现精确空间引导。我们在高分辨率3D医学图像合成任务上评估模型,对比基于U-Net的先进3D潜空间扩散模型。结果表明,模型在全局连贯性、生成保真度和可控性方面均有提升。研究发现,全Transformer扩散模型为体积医学图像合成提供了灵活基础。代码与在公开数据上训练的模型已开源:https://github.com/Cardio-AI/voldit。

原文摘要 · Abstract (English)

Diffusion models have become a leading approach for high-fidelity medical image synthesis. However, most existing methods for 3D medical image generation rely on convolutional U-Net backbones within latent diffusion frameworks. While effective, these architectures impose strong locality biases and limited receptive fields, which may constrain scalability, global context integration, and flexible conditioning. In this work, we introduce VolDiT, the first purely transformer-based 3D Diffusion Transformer for volumetric medical image synthesis. Our approach extends diffusion transformers to native 3D data through volumetric patch embeddings and global self-attention operating directly over 3D tokens. To enable structured control, we propose a timestep-gated control adapter that maps segmentation masks into learnable control tokens that modulate transformer layers during denoising. This token-level conditioning mechanism allows precise spatial guidance while preserving the modeling advantages of transformer architectures. We evaluate our model on high-resolution 3D medical image synthesis tasks and compare it to state-of-the-art 3D latent diffusion models based on U-Nets. Results demonstrate improved global coherence, superior generative fidelity, and enhanced controllability. Our findings suggest that fully transformerbased diffusion models provide a flexible foundation for volumetric medical image synthesis. The code and models trained on public data are available at https://github.com/Cardio-AI/voldit.

3D生成扩散模型Transformer医学影像

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。