用语义知识增强SAM,提升医学图像分割精度。
SEG-SAM: Semantic-Guided SAM for Unified Medical Image Segmentation
- 引入独立语义解码器,分离二值与语义分割任务。
- 融合大模型提取的医学特征,提升类别理解能力。
- 跨掩码空间对齐策略,增强双解码器预测一致性。
近期,统一医学图像分割模型受到广泛关注,尤其在视觉基础模型如通用图像分割模型(SAM)出现后。尽管SAM在自然图像中表现出色,但在医学领域仍面临挑战,因医学图像存在大量类别间重叠。为此,本文提出语义引导的SAM(SEG-SAM),通过整合医学语义知识提升分割性能。首先,为避免二值与语义预测间的冲突,设计一个独立于原SAM解码器的语义感知解码器,专门处理提示目标的语义分割及未提示对象的分类。其次,利用大语言模型提取医学类别关键特征,并通过文本到视觉语义模块,将语言信息自适应地融入视觉分割任务。最后,引入跨掩码空间对齐策略,促进两个解码器预测掩码间的重叠,从而协同优化二者结果。大量实验表明,SEG-SAM在统一二值医学分割上优于现有基于SAM的方法,在特定任务语义分割上亦超越当前最优方法,展现出广阔应用前景。
原文摘要 · Abstract (English)
Recently, developing unified medical image segmentation models gains increasing attention, especially with the advent of the Segment Anything Model (SAM). SAM has shown promising binary segmentation performance in natural domains, however, transferring it to the medical domain remains challenging, as medical images often possess substantial inter-category overlaps. To address this, we propose the SEmantic-Guided SAM (SEG-SAM), a unified medical segmentation model that incorporates semantic medical knowledge to enhance medical segmentation performance. First, to avoid the potential conflict between binary and semantic predictions, we introduce a semantic-aware decoder independent of SAM's original decoder, specialized for both semantic segmentation on the prompted object and classification on unprompted objects in images. To further enhance the model's semantic understanding, we solicit key characteristics of medical categories from large language models and incorporate them into SEG-SAM through a text-to-vision semantic module, adaptively transferring the language information into the visual segmentation task. In the end, we introduce the cross-mask spatial alignment strategy to encourage greater overlap between the predicted masks from SEG-SAM's two decoders, thereby benefiting both predictions. Extensive experiments demonstrate that SEG-SAM outperforms state-of-the-art SAM-based methods in unified binary medical segmentation and task-specific methods in semantic medical segmentation, showcasing promising results and potential for broader medical applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。