arXiv:2608.04676cs.CV2026-08

用生成式检索提升手术视频理解,速度快且准确。

SurgNarrator: A Generative Retrieval Framework for Surgical Video Understanding

论文配图:SurgNarrator: A Generative Retrieval Framework for Surgical Video Understanding
图 1 · 摘自论文原文
  • 构建手术专属词汇库,定义临床有意义的检索空间。
  • 零样本下在12个基准上优于现有方法,推理延迟降低90%以上。
  • 适合需要实时决策支持的手术场景,尤其适合临床部署。

手术过程由结构化且重复的临床事件构成,通过术中视频实现实时理解对术中决策至关重要。现有方法存在权衡:自回归视频-语言模型虽能全面推理但不适用于时间敏感的临床场景,对比模型延迟低却难以理解复杂场景。生成式检索虽在通用领域有探索,但移植至手术领域面临挑战——外观相近的视频可能语义不同,且术语高度专业化。为此,我们提出SurgNarrator,一种专为手术视频理解设计的生成式检索框架。通过手术描述构建精细的手术中心词汇库,定义临床有意义的检索空间;并基于预训练Qwen3-VL-Embedding-8B,采用时序感知对比目标学习判别性临床表征。推理时采用分层、流程感知的检索策略,缩小搜索范围至相关手术类型,实现快速高效响应。在十二个基准上以零样本设置全面评估,性能持续优于当前最优基线,输出阶段延迟较生成基线降低两个数量级以上。

原文摘要 · Abstract (English)

Surgical procedures unfold as structured and recurring clinical events, whose real-time understanding via intraoperative surgical videos is critical for intraoperative decision-making and support. However, existing video understanding methods force a trade-off: autoregressive video-language models support comprehensive reasoning but are not practical for time-sensitive clinical applications, whereas contrastive models offer low latency but struggle with complex scene understanding. Recently, generative retrieval has been explored for general-domain video understanding, but transferring it to surgery is not trivial because near-identical visual appearances may indicate semantically distinct events, and the terminology involved is highly surgery-specific. To this end, we propose SurgNarrator, a new generative retrieval framework tailored for surgical video understanding. We construct a well-curated surgery-centric vocabulary from surgical captions to define a clinically meaningful retrieval space. We then adapt the pre-trained Qwen3-VL-Embedding-8B to learn discriminative clinical representations with a temporally-aware contrastive objective. During inference, a hierarchical, procedure-aware retrieval strategy narrows the search space to the relevant procedure type, delivering fast and effective responses. Our method is comprehensively evaluated on twelve benchmarks in a zero-shot setting and achieves consistent performance gains over state-of-the-art baselines, while reducing output-stage latency by more than two orders of magnitude compared with the generative baseline.

手术视频生成检索多模态实时分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。