用文本描述提升多器官分割精度,解决依赖几何提示和细节丢失问题
CRISP-SAM2: SAM2 with Cross-Modal Interaction and Semantic Prompting for Multi-Organ Segmentation
- 通过跨模态交互将图文信息融合为上下文语义
- 在7个公开数据集上平均性能超越现有模型
- 无需几何提示,适合临床医生用自然语言指导分割
多器官医学图像分割是医疗影像处理的关键环节,对精准诊断和治疗方案制定至关重要。尽管该领域进展显著,现有模型仍常存在细节不准确、依赖几何提示和空间信息丢失等问题。为此,我们提出基于SAM2的CRISP-SAM2模型,引入跨模态交互与语义提示机制。该方法通过渐进式跨注意力机制将视觉与文本输入转换为跨模态上下文语义,并注入图像编码器以增强视觉细节理解。为消除对几何提示的依赖,采用语义提示策略替代原始提示编码器,强化对困难目标的感知。此外,结合相似性排序的自更新记忆机制与掩码精修流程,进一步适配医学影像并提升局部细节表现。在七个公开数据集上的对比实验表明,CRISP-SAM2性能优于现有模型。大量分析验证了方法有效性,尤其在解决前述局限方面表现突出。代码已开源:https://github.com/YU-deep/CRISP_SAM2.git。
原文摘要 · Abstract (English)
Multi-organ medical segmentation is a crucial component of medical image processing, essential for doctors to make accurate diagnoses and develop effective treatment plans. Despite significant progress in this field, current multi-organ segmentation models often suffer from inaccurate details, dependence on geometric prompts and loss of spatial information. Addressing these challenges, we introduce a novel model named CRISP-SAM2 with CRoss-modal Interaction and Semantic Prompting based on SAM2. This model represents a promising approach to multi-organ medical segmentation guided by textual descriptions of organs. Our method begins by converting visual and textual inputs into cross-modal contextualized semantics using a progressive cross-attention interaction mechanism. These semantics are then injected into the image encoder to enhance the detailed understanding of visual information. To eliminate reliance on geometric prompts, we use a semantic prompting strategy, replacing the original prompt encoder to sharpen the perception of challenging targets. In addition, a similarity-sorting self-updating strategy for memory and a mask-refining process is applied to further adapt to medical imaging and enhance localized details. Comparative experiments conducted on seven public datasets indicate that CRISP-SAM2 outperforms existing models. Extensive analysis also demonstrates the effectiveness of our method, thereby confirming its superior performance, especially in addressing the limitations mentioned earlier. Our code is available at: https://github.com/YU-deep/CRISP_SAM2.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。