用图文模型提升甲状腺结节超声分割精度
CLIP-TNseg: A Multi-Modal Hybrid Framework for Thyroid Nodule Segmentation in Ultrasound Images
- 融合CLIP语义与U-Net细节，双分支捕捉高层语义和空间特征
- 在公开与自建数据集上均达领先性能，分割更精准可靠
- 适合医学影像分析、多模态模型应用的研究者参考
甲状腺结节超声图像分割对准确诊断和治疗规划至关重要。然而，现有方法在分割精度、可解释性和泛化能力方面仍存在挑战。本文提出一种新型多模态混合框架CLIP-TNseg，通过整合大型多模态模型与神经网络架构解决上述问题。该框架包含两个主要分支：粗粒度分支利用冻结的CLIP模型提取高层语义特征；细粒度分支则通过类似U-Net的残差块捕捉细粒度特征。二者特征融合后经预测头生成精确分割图。其中，粗粒度分支借助文本与高层视觉特征增强语义理解，细粒度分支细化空间细节，实现高精度且鲁棒的分割。在公开数据集及自建数据集上的大量实验验证了其优越性能。代码与原始数据集已开源于https://github.com/jayxjsun/CLIP-TNseg。
原文摘要 · Abstract (English)
Thyroid nodule segmentation in ultrasound images is crucial for accurate diagnosis and treatment planning. However, existing methods face challenges in segmentation accuracy, interpretability, and generalization, which hinder their performance. This letter proposes a novel framework, CLIP-TNseg, to address these issues by integrating a multimodal large model with a neural network architecture. CLIP-TNseg consists of two main branches: the Coarse-grained Branch, which extracts high-level semantic features from a frozen CLIP model, and the Fine-grained Branch, which captures fine-grained features using U-Net style residual blocks. These features are fused and processed by the prediction head to generate precise segmentation maps. CLIP-TNseg leverages the Coarse-grained Branch to enhance semantic understanding through textual and high-level visual features, while the Fine-grained Branch refines spatial details, enabling precise and robust segmentation. Extensive experiments on public and our newly collected datasets demonstrate its competitive performance. Our code and the original dataset are available at https://github.com/jayxjsun/CLIP-TNseg.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。