实时生成手术视频解说,精准捕捉流程变化。
SurgOnAir: Hierarchy-Aware Real-Time Surgical Video Commentary

- 逐帧处理视觉输入,边看边生成解说。
- 支持动作/步骤/阶段多层级描述,准确率超基线12.3%。
- 适合手术辅助系统开发与医学教育场景。
实时理解手术流程对智能手术系统至关重要,但现有方法多为离线处理或粗粒度分析,难以及时响应手术中细微变化。本文提出SurgOnAir,一种流式视觉语言模型,可逐帧处理视频并实时生成叙述性文本,实现帧级到令牌级的细粒度响应。基于自建的包含11,000样本的分层数据集SurgOnAir-11k,模型在动作、步骤和阶段三个层次上进行监督训练,能生成具有层次结构的多级文本输出。特别设计的转换标记可显式标识关键流程切换,实现对动态变化的即时捕捉。实验表明,该模型通过单一架构统一了多层级流式理解,生成的解说更具层次性和准确性,在多个指标上优于现有方法。代码与数据集将公开。
原文摘要 · Abstract (English)
Understanding surgical workflow in real time is fundamental for intelligent surgical embodiment, where AI systems continuously perceive and respond as surgery proceeds. In the operating room, critical decisions depend on subtle, moment-to-moment changes, such as fine instrument movements and evolving tissue states, where even slight perceptual delays can limit assistance or compromise safety. Yet existing methods remain offline or operate at coarse temporal scales, generating descriptions only after processing clips, preventing immediate reaction. We address this by proposing SurgOnAir, a streaming vision-language model that processes frames sequentially without future access and progressively generates narration tokens as visual input arrives. SurgOnAir achieves fine-grained frame-to-token generation, enabling instant responsiveness to evolving surgical dynamics. Built upon our curated hierarchical dataset SurgOnAir-11k spanning action-, step-, and phase-level supervision, the model is trained to produce multi-level textual responses that reflect the inherent hierarchy of surgical procedures. Furthermore, special transition tokens are generated to explicitly mark state changes, allowing SurgOnAir to capture and signal key workflow transitions as they occur. Experiments show that SurgOnAir enables real-time understanding through a single vision-language model that unifies streaming across multiple hierarchies of the surgical workflow, generating superior and hierarchy-aware narrations. Code and dataset will be public.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。