用大模型提升医学影像分割精度,实现跨模态通用
MedDINOv3: How to adapt vision foundation models for medical image segmentation?
- 设计多尺度特征聚合结构,优化ViT在医疗图像上的表现
- 在387万张腹部CT切片上进行分阶段预训练,提升模型泛化能力
- 在4个基准上超越或持平当前最优,适合医疗影像通用建模
准确分割CT和MRI中的器官与肿瘤对诊断、治疗规划和疾病监测至关重要。尽管深度学习已推动自动分割发展,但多数模型仍为特定任务设计,缺乏跨模态与机构的泛化能力。基于十亿级自然图像预训练的视觉基础模型(Vision Foundation Models, FMs)提供强大且可迁移的表征。然而,将其应用于医学影像面临两大挑战:(1) 多数基础模型的ViT骨干网络在医学图像分割任务上仍逊于专用CNN;(2) 自然图像与医学图像间存在显著领域差异,限制迁移效果。本文提出MedDINOv3,一种简单高效的DINOv3适配框架。首先重新审视普通ViT,设计一种多尺度令牌聚合结构;随后在包含387万张轴向CT切片的高质量数据集CT-3M上,采用多阶段DINOv3训练策略,学习鲁棒的密集特征表示。MedDINOv3在四个分割基准上达到或超过现有最佳性能,证明了视觉基础模型作为医学图像分割统一骨干网络的巨大潜力。代码开源于https://github.com/ricklisz/MedDINOv3。
原文摘要 · Abstract (English)
Accurate segmentation of organs and tumors in CT and MRI scans is essential for diagnosis, treatment planning, and disease monitoring. While deep learning has advanced automated segmentation, most models remain task-specific, lacking generalizability across modalities and institutions. Vision foundation models (FMs) pretrained on billion-scale natural images offer powerful and transferable representations. However, adapting them to medical imaging faces two key challenges: (1) the ViT backbone of most foundation models still underperform specialized CNNs on medical image segmentation, and (2) the large domain gap between natural and medical images limits transferability. We introduce MedDINOv3, a simple and effective framework for adapting DINOv3 to medical segmentation. We first revisit plain ViTs and design a simple and effective architecture with multi-scale token aggregation. Then, we perform domain-adaptive pretraining on CT-3M, a curated collection of 3.87M axial CT slices, using a multi-stage DINOv3 recipe to learn robust dense features. MedDINOv3 matches or exceeds state-of-the-art performance across four segmentation benchmarks, demonstrating the potential of vision foundation models as unified backbones for medical image segmentation. The code is available at https://github.com/ricklisz/MedDINOv3.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。