用文本提示提升医学图像病灶分割精度，无需推理时输入文本
STPNet: Scale-aware Text Prompt Network for Medical Image Segmentation
- 通过多尺度文本描述引导病灶定位，融合视觉与语言信息
- 在三个数据集上超越现有方法，最高提升4.2%的Dice分数
- 训练时自动检索医疗文本，推理时无需输入，适合临床部署
准确分割病变对医学影像分析和诊断至关重要。传统仅依赖视觉特征的方法在病灶分布和大小不确定性上表现不佳。为此，我们提出STPNet——一种感知尺度的文本提示网络，利用视觉-语言建模增强医学图像分割。该方法采用多尺度文本描述指导病灶定位，并通过检索-分割联合学习弥合视觉与语言模态间的语义鸿沟。关键在于，STPNet在训练阶段从专用医疗文本库中检索相关文本信息，推理时无需输入文本，同时保留跨模态学习优势。我们在COVID-Xray、COVID-CT和Kvasir-SEG三个数据集上评估，实验结果表明，该视觉-语言方法优于现有先进分割模型，证明了引入文本语义知识的有效性。代码已公开于https://github.com/HUANGLIZI/STPNet。
原文摘要 · Abstract (English)
Accurate segmentation of lesions plays a critical role in medical image analysis and diagnosis. Traditional segmentation approaches that rely solely on visual features often struggle with the inherent uncertainty in lesion distribution and size. To address these issues, we propose STPNet, a Scale-aware Text Prompt Network that leverages vision-language modeling to enhance medical image segmentation. Our approach utilizes multi-scale textual descriptions to guide lesion localization and employs retrieval-segmentation joint learning to bridge the semantic gap between visual and linguistic modalities. Crucially, STPNet retrieves relevant textual information from a specialized medical text repository during training, eliminating the need for text input during inference while retaining the benefits of cross-modal learning. We evaluate STPNet on three datasets: COVID-Xray, COVID-CT, and Kvasir-SEG. Experimental results show that our vision-language approach outperforms state-of-the-art segmentation methods, demonstrating the effectiveness of incorporating textual semantic knowledge into medical image analysis. The code has been made publicly on https://github.com/HUANGLIZI/STPNet.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。