arXiv:2511.03325cs.CV2025-11被引 7

让AI理解手术视频动态过程,回答更准确

SurgViVQA: Temporally-Grounded Video Question Answering for Surgical Scene Understanding

  • 用视频-文本联合编码器捕捉运动和器械互动等时序信息
  • 在两个数据集上关键词准确率比现有模型高9%-11%
  • 适合需要理解手术动态过程的研究者和临床辅助系统开发者

外科视频问答(VideoQA)旨在通过让AI模型推理时间连贯事件而非孤立帧,提升术中理解能力。当前方法多依赖静态图像特征,且可用数据集常缺乏时序标注,忽略对准确流程解读至关重要的动态信息。本文提出SurgViVQA,一种将视觉推理从静态图像扩展到动态手术场景的模型。它采用掩码视频-文本编码器融合视频与问题特征,捕捉运动、器械-组织交互等时序线索,并由微调的大语言模型生成连贯答案。为评估性能,我们构建了REAL-Colon-VQA数据集,包含与运动相关的问题、诊断属性,以及重述或语义变化的非模板问题,以测试模型鲁棒性。在REAL-Colon-VQA和公开的EndoVis18-VQA数据集上的实验表明,SurgViVQA优于现有基于图像的VQA基准模型,关键词准确率分别提升11%和9%。对问题扰动的分析进一步验证了其更强的泛化能力和对提问方式变化的鲁棒性。SurgViVQA与REAL-Colon-VQA共同构成一个面向手术视频的时序感知理解框架,使AI能更有效地解析动态操作上下文。代码与数据集见https://github.com/madratak/SurgViVQA。

原文摘要 · Abstract (English)

Video Question Answering (VideoQA) in the surgical domain aims to enhance intraoperative understanding by enabling AI models to reason over temporally coherent events rather than isolated frames. Current approaches are limited to static image features, and available datasets often lack temporal annotations, ignoring the dynamics critical for accurate procedural interpretation. We propose SurgViVQA, a surgical VideoQA model that extends visual reasoning from static images to dynamic surgical scenes. It uses a Masked Video--Text Encoder to fuse video and question features, capturing temporal cues such as motion and tool--tissue interactions, which a fine-tuned large language model (LLM) then decodes into coherent answers. To evaluate its performance, we curated REAL-Colon-VQA, a colonoscopic video dataset that includes motion-related questions and diagnostic attributes, as well as out-of-template questions with rephrased or semantically altered formulations to assess model robustness. Experimental validation on REAL-Colon-VQA and the public EndoVis18-VQA dataset shows that SurgViVQA outperforms existing image-based VQA benchmark models, particularly in keyword accuracy, improving over PitVQA by +11\% on REAL-Colon-VQA and +9\% on EndoVis18-VQA. A perturbation study on the questions further confirms improved generalizability and robustness to variations in question phrasing. SurgViVQA and the REAL-Colon-VQA dataset provide a framework for temporally-aware understanding in surgical VideoQA, enabling AI models to interpret dynamic procedural contexts more effectively. Code and dataset available at https://github.com/madratak/SurgViVQA.

视频问答手术理解时序建模大模型应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。