构建动态手术场景图,提升视觉语言模型的时序推理能力
STSG-VQA: Evidence-Grounded Temporal Question Answering from Surgical Spatio-Temporal Scene Graphs

- 用多层级时序监督增强手术场景图,捕捉物体、事件与流程的动态变化
- 在18,458个问题上提升模型准确率,最高超基线24.39个百分点
- 适合医疗AI、手术分析与可解释视觉语言模型研究者
尽管手术视觉语言模型(VLM)取得进展,但时序推理仍受限于以帧为中心的标注。现有帧级场景图虽能结构化表示手术环境,却未显式建模手术流程的动态演变。为此,本文提出多层级时序监督方法,将对象级连续性、事件级交互连续性和流程级连通性融入帧级手术场景图,构建时空场景图(STSG)。基于此,执行时序问答生成有证据支持的答案对,形成STSG-VQA基准。每个问题关联其时间区间与支撑证据,实现可追溯验证。基准涵盖7类时序问题,共18,458个问答对。使用STSG监督微调Qwen3-VL-4B和Hulu-Med-4B,相比零样本基线分别提升24.39和19.56个百分点,较静态场景图监督提升16.50和14.25个百分点,且各类时序任务均获增益,表明该方法有效促进模型对时序交互的理解。代码与数据集将在论文接受后公开。
原文摘要 · Abstract (English)
Despite recent advances in surgical vision-language models (VLMs), temporal reasoning remains limited because existing supervision is largely frame-centric. Frame-level scene graphs (SGs) have proven effective in providing structured representations of surgical environments but do not explicitly model the dynamics of surgical workflows. To explicitly model how surgical states evolve across time, we introduce a multi-level structured temporal supervision methodology that augments frame-level surgical SGs with object-level continuity, event-level interaction continuity, and procedure-level connectivity. We then execute temporal queries over the resulting spatio-temporal scene graphs (STSGs) to generate evidence-grounded question-answer pairs, which together form the STSG-VQA benchmark. Each question is linked to the temporal interval and STSG evidence used to derive its reference answer, enabling traceable verification. The benchmark contains 18,458 question-answer pairs across seven temporal categories. Fine-tuning Qwen3-VL-4B and Hulu-Med-4B with STSG-derived supervision improves question-level micro accuracy by 24.39 and 19.56 percentage points over their zero-shot baselines and by 16.50 and 14.25 points over static scene-graph supervision, respectively. These gains span all temporal categories, indicating that STSG-derived supervision helps surgical VLMs reason over temporally grounded interactions rather than isolated frames. The code and dataset will be made publicly available upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。