arXiv:2604.03645eess.IVcs.CV2026-04

统一支持视觉/文本/语音提示的手术视频分割模型,提升实时可靠性。

UniSurgSAM: A Unified Promptable Model for Reliable Surgical Video Segmentation

  • 分阶段解耦设计,分别优化目标初始化与跟踪,避免干扰。
  • 三种机制抑制幻觉、防止遮挡漂移、实现失败自恢复。
  • 首个多模态多粒度手术视频分割基准,支持真实场景应用。

手术视频分割是计算机辅助手术的基础。实际中,外科医生需在长时间手术过程中动态指定目标,使用视觉选择、文本描述或语音指令等异构提示。现有可提示视频目标分割(PVOS)方法通常仅支持单一提示模态,且依赖耦合框架,导致初始化与跟踪间存在优化干扰。此外,当目标消失时会产生幻觉预测,且缺乏故障恢复能力,易出现累积遮挡漂移。为此,我们提出UniSurgSAM,一种统一的可提示模型,支持视觉、文本或语音提示,实现可靠的手术视频分割。具体而言,采用解耦两阶段框架,独立优化初始化与跟踪以解决优化干扰问题。在此框架内,引入三项关键设计:感知存在性解码,建模目标缺失以抑制幻觉;边界感知长时跟踪,防止长时间序列中的遮挡漂移;自适应状态转换,实现阶段间闭环反馈以支持失败恢复。同时,基于四个公开手术数据集构建了多模态、多粒度的基准,包含精细实例级掩码片段。大量实验表明,UniSurgSAM在所有提示模态和粒度下均达到实时状态领先性能,为计算机辅助手术提供了实用基础。代码与数据集将发布于 https://jinlab-imvr.github.io/UniSurgSAM。

原文摘要 · Abstract (English)

Surgical video segmentation is fundamental to computer-assisted surgery. In practice, surgeons need to dynamically specify targets throughout extended procedures, using heterogeneous cues such as visual selections, textual expressions, or audio instructions. However, existing Promptable Video Object Segmentation (PVOS) methods are typically restricted to a single prompt modality and rely on coupled frameworks that cause optimization interference between target initialization and tracking. Moreover, these methods produce hallucinated predictions when the target is absent and suffer from accumulated mask drift without failure recovery. To address these challenges, we present UniSurgSAM, a unified PVOS model enabling reliable surgical video segmentation through visual, textual, or audio prompts. Specifically, UniSurgSAM employs a decoupled two-stage framework that independently optimizes initialization and tracking to resolve the optimization interference. Within this framework, we introduce three key designs for reliability: presence-aware decoding that models target absence to suppress hallucinations; boundary-aware long-term tracking that prevents mask drift over extended sequences; and adaptive state transition that closes the loop between stages for failure recovery. Furthermore, we establish a multi-modal and multi-granular benchmark from four public surgical datasets with precise instance-level masklets. Extensive experiments demonstrate that UniSurgSAM achieves state-of-the-art performance in real time across all prompt modalities and granularities, providing a practical foundation for computer-assisted surgery. Code and datasets will be available at https://jinlab-imvr.github.io/UniSurgSAM.

视频分割手术辅助多模态实时推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。