用文字提示实现无需标注的手术器械分割,突破传统依赖人工标注的瓶颈。
Toward Mask Annotation-Free Surgical Instrument Segmentation from Endoscopic Images Using Text-Prompted Segment Anything Model 3 (SAM3)

- 先用通用提示'tool'生成二值掩码,再通过视觉语言模型区分具体器械
- 在EndoVis数据集上优于直接使用SAM3文本提示,实例分割效果提升显著
- 适合追求自动化、减少标注成本的医疗AI研究者参考
手术器械分割是计算机辅助干预的基础任务,但现有方法多依赖像素级标注或手动空间提示,限制了可扩展性和自动化。新提出的Segment Anything Model 3(SAM3)可通过文本提示实现无标注自动分割,但因领域差异,直接使用器械名称作为提示效果不佳。为此,我们提出两阶段框架:第一阶段利用与自然语言对齐的通用提示‘tool’,借助SAM3零样本能力生成二值掩码;第二阶段通过在SAM3生成掩码区域上微调的视觉-语言模型Qwen,实现器械实例级分类与分割。在EndoVis 2017和2018数据集上的评估表明,尽管性能未达全监督方法水平,但显著优于直接使用SAM3进行实例级分割。结果揭示了SAM3的潜力与局限,为无标注手术器械分割提供了可行方向。
原文摘要 · Abstract (English)
Surgical instrument segmentation is a fundamental task for computer-assisted interventions, yet most existing methods rely on pixel-level annotations or manual spatial prompts, which limit scalability and automation. The recently introduced Segment Anything Model 3 (SAM3) offers a pathway to annotation-free, automatic segmentation via text-based prompting; however, the instrument name as a text prompt could not be directly used due to a large domain gap. To overcome these limitations, we propose a two-stage framework that achieves instance-level segmentation without requiring ground truth masks or manual interaction. In the first stage, we leverage a natural-language-aligned generic prompt - "tool" - to produce binary masks using SAM3's zero-shot capability. In the second stage, these masks are extended to instance-level by integrating a vision-language model (Qwen) that is fine-tuned on SAM3-generated masked regions for instrument classification. We evaluate our approach on the EndoVis 2017 and 2018 datasets. Results show that, while our two-stage approach does not reach the performance of current fully supervised methods, it significantly outperforms the direct use of SAM3 for instance-level instrument segmentation with text prompts. Overall, our findings highlight both the limitations and potential of SAM3, suggesting a promising direction toward annotation-free surgical instrument segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。