arXiv:2505.08581cs.CVeess.IV2025-05中稿 · MICCAI 2025被引 19

ReSurgSAM2实现手术视频中目标的实时精准追踪,支持文本指代。

ReSurgSAM2: Referring Segment Anything in Surgical Video via Credible Long-term Tracking

  • 两阶段框架:先用跨模态时空Mamba检测目标,再可靠初始化追踪
  • 在真实手术视频上实现61.2帧/秒的实时追踪,准确率显著提升
  • 适合临床辅助手术系统开发人员,尤其关注长期精准追踪场景

手术场景分割对计算机辅助手术至关重要,有助于提升手术质量与患者预后。近年来,基于文本指代的手术分割方法兴起,为外科医生提供交互式目标分割能力。然而,现有方法受限于低效与短期追踪,难以应用于复杂真实的手术场景。本文提出ReSurgSAM2,一种两阶段手术指代分割框架:首先利用Segment Anything Model 2进行文本指代目标检测,随后通过可信初始帧识别与多样性驱动的长期记忆机制实现稳定追踪。检测阶段引入跨模态时空Mamba,生成精确检测与分割结果;基于该结果,采用可信初始帧选择策略确定可靠追踪起始帧;追踪阶段则构建兼具可信性与多样性的记忆库,确保长期一致性。大量实验表明,ReSurgSAM2相比现有方法在准确率与效率上均有显著提升,实现实时运行(61.2 FPS)。代码与数据集将开源于https://github.com/jinlab-imvr/ReSurgSAM2。

原文摘要 · Abstract (English)

Surgical scene segmentation is critical in computer-assisted surgery and is vital for enhancing surgical quality and patient outcomes. Recently, referring surgical segmentation is emerging, given its advantage of providing surgeons with an interactive experience to segment the target object. However, existing methods are limited by low efficiency and short-term tracking, hindering their applicability in complex real-world surgical scenarios. In this paper, we introduce ReSurgSAM2, a two-stage surgical referring segmentation framework that leverages Segment Anything Model 2 to perform text-referred target detection, followed by tracking with reliable initial frame identification and diversity-driven long-term memory. For the detection stage, we propose a cross-modal spatial-temporal Mamba to generate precise detection and segmentation results. Based on these results, our credible initial frame selection strategy identifies the reliable frame for the subsequent tracking. Upon selecting the initial frame, our method transitions to the tracking stage, where it incorporates a diversity-driven memory mechanism that maintains a credible and diverse memory bank, ensuring consistent long-term tracking. Extensive experiments demonstrate that ReSurgSAM2 achieves substantial improvements in accuracy and efficiency compared to existing methods, operating in real-time at 61.2 FPS. Our code and datasets will be available at https://github.com/jinlab-imvr/ReSurgSAM2.

手术分割长时追踪多模态实时系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。