用文字描述替代点击,实现医学图像零样本分割
DescriptorMedSAM: Language-Image Fusion with Multi-Aspect Text Guidance for Medical Image Segmentation
- 引入结构化文本提示,融合名称、形状、位置信息
- 零样本下Dice达0.9405,微调仅需50张标注图
- 适合医疗场景中少样本快速部署的分割需求
精准器官分割对放疗规划和疾病监测至关重要。现有基础模型如MedSAM虽表现优异,但仍需人工交互。我们提出DescriptorMedSAM,一种轻量级MedSAM扩展模型,通过结构化文本提示(从器官名称到包含形状与位置的复合描述)实现无需点击的分割。该模型采用CLIP文本编码器将放射科风格描述转为密集嵌入,经交叉注意力块与多尺度特征提取器融合视觉特征。设计四类提示:名称(N)、名称+形状(NS)、名称+位置(NL)、名称+形状+位置(NSL),在FLARE 2022数据集上评估其零样本与少样本性能。其中NSL提示表现最优,在全监督下Dice达0.9405,零样本保留率76.31%,仅需每未见器官50张标注切片微调后保留率达97.02%。加入形状与位置信息显著提升小或形态复杂结构的分割精度。结果表明,结构化语言提示可有效替代空间交互,兼具强零样本性能与快速少样本适应能力。本工作量化了提示类型的作用,为低注释成本的可扩展、提示感知分割模型奠定基础。
原文摘要 · Abstract (English)
Accurate organ segmentation is essential for clinical tasks such as radiotherapy planning and disease monitoring. Recent foundation models like MedSAM achieve strong results using point or bounding-box prompts but still require manual interaction. We propose DescriptorMedSAM, a lightweight extension of MedSAM that incorporates structured text prompts, ranging from simple organ names to combined shape and location descriptors to enable click-free segmentation. DescriptorMedSAM employs a CLIP text encoder to convert radiology-style descriptors into dense embeddings, which are fused with visual tokens via a cross-attention block and a multi-scale feature extractor. We designed four descriptor types: Name (N), Name + Shape (NS), Name + Location (NL), and Name + Shape + Location (NSL), and evaluated them on the FLARE 2022 dataset under zero-shot and few-shot settings, where organs unseen during training must be segmented with minimal additional data. NSL prompts achieved the highest performance, with a Dice score of 0.9405 under full supervision, a 76.31% zero-shot retention ratio, and a 97.02% retention ratio after fine-tuning with only 50 labeled slices per unseen organ. Adding shape and location cues consistently improved segmentation accuracy, especially for small or morphologically complex structures. We demonstrate that structured language prompts can effectively replace spatial interactions, delivering strong zero-shot performance and rapid few-shot adaptation. By quantifying the role of descriptor, this work lays the groundwork for scalable, prompt-aware segmentation models that generalize across diverse anatomical targets with minimal annotation effort.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。