用大模型知识蒸馏提升小模型肠镜息肉分割精度,效率提升近9倍。
From SAM to DINOv2: Towards Distilling Foundation Models to Lightweight Baselines for Generalized Polyp Segmentation
- 将大模型语义先验注入U-Net等轻量架构,结合频域编码增强蒸馏效果
- 在5个数据集上超越基线与当前最优模型,计算量减少近9倍
- 适合医疗影像部署场景,兼顾精度与实时性
肠镜中精准分割息肉对结直肠癌早期发现至关重要,但因大小、形状、颜色差异大且易伪装,仍具挑战。轻量级模型如U-Net、U-Net++和PraNet虽易于部署、计算成本低,但难以应对上述问题,性能有限。相比之下,SAM、DINOv2等大规模视觉基础模型在自然图像领域表现出色。然而其直接迁移至医学影像任务(如肠镜息肉分割)面临数据稀缺与领域知识缺失难题。为此,我们提出Polyp-DiFoM蒸馏框架,将基础模型的丰富表征迁移到轻量分割基线中,实现临床环境下的高效高精度部署。具体通过注入基础模型语义先验至U-Net、U-Net++等结构,并引入频域编码以增强蒸馏效果,验证了其泛化能力。在Kvasir-SEG、CVC-ClinicDB、ETIS、ColonDB、CVC-300五个基准数据集上广泛实验表明,Polyp-DiFoM显著优于对应基线及当前最优模型,计算开销近乎降低9倍。代码已公开于https://github.com/lostinrepo/PolypDiFoM。
原文摘要 · Abstract (English)
Accurate polyp segmentation during colonoscopy is critical for the early detection of colorectal cancer and still remains challenging due to significant size, shape, and color variations, and the camouflaged nature of polyps. While lightweight baseline models such as U-Net, U-Net++, and PraNet offer advantages in terms of easy deployment and low computational cost, they struggle to deal with the above issues, leading to limited segmentation performance. In contrast, large-scale vision foundation models such as SAM, DINOv2, OneFormer, and Mask2Former have exhibited impressive generalization performance across natural image domains. However, their direct transfer to medical imaging tasks (e.g., colonoscopic polyp segmentation) is not straightforward, primarily due to the scarcity of large-scale datasets and lack of domain-specific knowledge. To bridge this gap, we propose a novel distillation framework, Polyp-DiFoM, that transfers the rich representations of foundation models into lightweight segmentation baselines, allowing efficient and accurate deployment in clinical settings. In particular, we infuse semantic priors from the foundation models into canonical architectures such as U-Net and U-Net++ and further perform frequency domain encoding for enhanced distillation, corroborating their generalization capability. Extensive experiments are performed across five benchmark datasets, such as Kvasir-SEG, CVC-ClinicDB, ETIS, ColonDB, and CVC-300. Notably, Polyp-DiFoM consistently outperforms respective baseline models significantly, as well as the state-of-the-art model, with nearly 9 times reduced computation overhead. The code is available at https://github.com/lostinrepo/PolypDiFoM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。