arXiv:2511.12098cs.CV2025-11

用自监督Transformer指导医学影像生成,实现更精准的合成CT。

DINOv3-Guided Cross Fusion Framework for Semantic-aware CT generation from MRI and CBCT

  • 融合冻结的DINOv3全局语义与CNN局部特征,分层交叉融合。
  • 在盆腔数据集上,合成CT的MS-SSIM和PSNR均达当前最优。
  • 首次将DINOv3用于医疗图像转换,适合放射治疗研究者参考。

从CBCT或MRI生成合成CT图像,可有效支持辐射剂量规划与自适应放疗。然而,现有基于CNN的模型缺乏全局语义理解,而Transformer因参数量大且归纳偏置弱,易在小样本医疗数据上过拟合。为此,我们提出DINOv3引导的跨模态融合框架(DGCF),将冻结的自监督DINOv3 Transformer与可训练的CNN编码器-解码器结合,通过可学习的跨融合模块,分层融合Transformer的全局表征与CNN的局部特征,实现局部外观与上下文语义的平衡表达。此外,引入多层级DINOv3感知损失(MLDP),促使合成CT与真实CT在DINOv3特征空间中具备语义相似性。在SynthRAD2023盆腔数据集上的实验表明,DGCF在MRI→CT与CBCT→CT任务中,于MS-SSIM、PSNR及基于分割的指标上均达到当前最优表现。据我们所知,这是首个将DINOv3表征用于医学图像翻译的工作,凸显了自监督Transformer在语义感知合成CT中的潜力。代码已开源:https://github.com/HiLab-git/DGCF。

原文摘要 · Abstract (English)

Generating synthetic CT images from CBCT or MRI has a potential for efficient radiation dose planning and adaptive radiotherapy. However, existing CNN-based models lack global semantic understanding, while Transformers often overfit small medical datasets due to high model capacity and weak inductive bias. To address these limitations, we propose a DINOv3-Guided Cross Fusion (DGCF) framework that integrates a frozen self-supervised DINOv3 Transformer with a trainable CNN encoder-decoder. It hierarchically fuses global representation of Transformer and local features of CNN via a learnable cross fusion module, achieving balanced local appearance and contextual representation. Furthermore, we introduce a Multi-Level DINOv3 Perceptual (MLDP) loss that encourages semantic similarity between synthetic CT and the ground truth in DINOv3's feature space. Experiments on the SynthRAD2023 pelvic dataset demonstrate that DGCF achieved state-of-the-art performance in terms of MS-SSIM, PSNR and segmentation-based metrics on both MRI$\rightarrow$CT and CBCT$\rightarrow$CT translation tasks. To the best of our knowledge, this is the first work to employ DINOv3 representations for medical image translation, highlighting the potential of self-supervised Transformer guidance for semantic-aware CT synthesis. The code is available at https://github.com/HiLab-git/DGCF.

医学图像生成Transformer跨模态融合自监督学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。