提升手术视频指代分割的准确性,通过可靠性引导锚点定位
ReGround-Surg: Reliability-Guided Anchor Grounding for Referring Surgical Video Segmentation

- 根据语义描述生成视觉可靠性图,指导锚点选择
- 在三个数据集上均优于现有方法,且推理速度几乎不变
- 适合需要高精度分割的医疗视频分析场景
指代手术视频分割需根据自然语言描述,在视频帧中分割目标器械或组织区域。基于SAM2的两阶段方法(如ReSurgSAM2)先在初始帧中定位目标,再通过追踪传播掩码。然而其性能高度依赖初始锚点质量:一旦选错,错误会持续传播。该问题在手术视频中尤为严重,因器械外观相似、遮挡及复杂组织-工具交互导致。为此,我们提出ReGround-Surg,一种轻量级可靠性引导锚点定位框架,以提升基于SAM2的指代手术视频分割性能。首先,从语言表达与当前帧视觉特征中预测文本条件的空间可靠性图。该图在两个互补分支中复用:门控侧适配器在图文融合前增强表达相关视觉区域;可靠性加权视觉到文本注意力模块在提示词聚合时抑制非目标视觉证据。在Ref-EndoVis17和Ref-EndoVis18上的实验表明,该方法在三个评估划分上均一致超越当前最优方法,且速度几乎无损耗。代码已公开于https://github.com/JiaxinWen1/ReGround-Surg。
原文摘要 · Abstract (English)
Referring surgical video segmentation requires segmenting a target instrument or tissue region across video frames according to a natural language expression. Recent Segment Anything Model 2 (SAM2) based two-stage methods (e.g., ReSurgSAM2) first ground the referred target in an initial or selected frame, then propagate the selected mask via tracking. Although effective, their performance is highly sensitive to the quality of the initial grounded mask: once an incorrect anchor is selected, subsequent tracking tends to propagate the error. This issue is especially challenging in surgical videos due to visually similar instruments, occlusion, and complex tissue-tool interactions. To address this issue, we propose ReGround-Surg, a lightweight reliability-guided anchor grounding framework to improve SAM2-based referring surgical video segmentation. It first predicts a text-conditioned spatial reliability map from the referring expression and current-frame visual features. The map is then reused in two complementary branches: a Gated Side Adapter enhances expression-relevant visual regions before text-to-vision fusion, while a Reliability-Weighted Vision-to-Text Attention module suppresses off-target visual evidence during prompt-token aggregation. Experiments on Ref-EndoVis17 and Ref-EndoVis18 show consistent improvements over state-of-the-art methods across three evaluation splits with negligible speed reduction. Code is publicly available at https://github.com/JiaxinWen1/ReGround-Surg.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。