arXiv:2511.03178cs.CV2025-11

首个面向手术未来事件预测的视觉问答模型,让系统提前预判下一步操作。

SurgAnt-ViVQA: Learning to Anticipate Surgical Events through GRU-Driven Temporal Cross-Attention

  • 用双向GRU捕捉视频帧间动态,通过门控机制融合视觉与语言信息
  • 在33.5小时手术视频上构建73万+问答对,实现未来阶段、器械、时长等多任务预测
  • 适合临床辅助系统研发者,推动手术智能助手从回溯描述转向主动预判

在鼻内经蝶垂体手术中,由于视野受限且流程变化迅速,预判即将发生的手术事件至关重要。现有视觉问答(VQA)系统多基于孤立帧进行静态跨模态对齐,难以支持未来步骤或器械需求的预测。为此,我们构建了首个面向前瞻性手术推理的视觉问答数据集PitVQA-Anticipation,包含33.5小时手术视频和734,769个问答对,覆盖未来阶段、下一步操作、即将使用器械及剩余时间四个任务。我们提出SurgAnt-ViVQA,一种基于大语言模型的视频语言模型,采用GRU门控时序交叉注意力模块:双向GRU编码帧间动态,自适应门控在词元层面注入视觉上下文。通过参数高效微调使语言主干适配手术领域。该模型在PitVQA-Anticipation与EndoVis数据集上超越强基线。消融实验表明,时序递归与门控融合带来主要性能提升。帧预算研究显示:8帧可最大化流畅性,32帧虽轻微降低BLEU但更优估计时间。通过时序感知编码器与细粒度门控交叉注意力结合,SurgAnt-ViVQA将手术VQA从回溯描述推进至主动预判。PitVQA-Anticipation为该场景提供全面基准,凸显精准时序建模对可靠未来感知手术辅助的重要性。

原文摘要 · Abstract (English)

Anticipating forthcoming surgical events is vital for real-time assistance in endonasal transsphenoidal pituitary surgery, where visibility is limited and workflow changes rapidly. Most visual question answering (VQA) systems reason on isolated frames with static vision language alignment, providing little support for forecasting next steps or instrument needs. Existing surgical VQA datasets likewise center on the current scene rather than the near future. We introduce PitVQA-Anticipation, the first VQA dataset designed for forward looking surgical reasoning. It comprises 33.5 hours of operative video and 734,769 question answer pairs built from temporally grouped clips and expert annotations across four tasks: predicting the future phase, next step, upcoming instrument, and remaining duration. We further propose SurgAnt-ViVQA, a video language model that adapts a large language model using a GRU Gated Temporal Cross-Attention module. A bidirectional GRU encodes frame to frame dynamics, while an adaptive gate injects visual context into the language stream at the token level. Parameter efficient fine tuning customizes the language backbone to the surgical domain. SurgAnt-ViVQA tested upon on PitVQA-Anticipation and EndoVis datasets, surpassing strong image and video based baselines. Ablations show that temporal recurrence and gated fusion drive most of the gains. A frame budget study indicates a trade-off: 8 frames maximize fluency, whereas 32 frames slightly reduce BLEU but improve numeric time estimation. By pairing a temporally aware encoder with fine grained gated cross-attention, SurgAnt-ViVQA advances surgical VQA from retrospective description to proactive anticipation. PitVQA-Anticipation offers a comprehensive benchmark for this setting and highlights the importance of targeted temporal modeling for reliable, future aware surgical assistance.

手术预测视觉问答时序建模医疗AI

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。