arXiv:2606.15861cs.CV2026-06

用对象令牌统一分割与手术视觉问答,提升精细场景理解。

Object Tokens as a Bridge Between Segmentation and Visual Question Answering in Robotic Surgery

论文配图:Object Tokens as a Bridge Between Segmentation and Visual Question Answering in Robotic Surgery
图 1 · 摘自论文原文
  • 用对象令牌连接视觉分割与语言推理,实现端到端联合建模。
  • 在RAMIE和EndoVis18数据集上,问答准确率优于基线方法。
  • 适合需要精准手术场景理解的医疗AI研究者和开发者。

机器人手术中的视觉问答(手术VQA)需对复杂手术场景进行高层理解,并融合视觉感知与语言推理,有望支持手术训练与术中决策。现有视觉语言模型虽通过参数高效微调展现潜力,但多数依赖粗粒度视觉定位(如边界框),难以捕捉手术物体的细粒度空间结构。本文提出一种统一框架,将像素级分割与视觉问答联合建模。该方法融合视觉语言模型(VLM)与基于Segment Anything Model(SAM)的解码器,将场景元素表示为由VLM生成的对象令牌。这些对象令牌既用于答案预测,又投影至SAM解码器生成分割掩码。通过同时优化分割与问答目标,模型学习到空间对齐的表示,增强视觉推理能力并提供明确的像素级定位。在私有RAMIE(机器人辅助微创食管切除术)和公开EndoVis18数据集上,该方法持续超越基线,在手术VQA任务中表现更优。结果表明,将上下文感知的对象令牌融入视觉语言模型,能显著提升手术场景的细粒度理解能力。

原文摘要 · Abstract (English)

Visual Question Answering (VQA) in robotic surgery, referred to as surgical VQA, requires high-level understanding of complex surgical scenes and the integration of visual perception with language reasoning, with the potential to support surgical training and intraoperative decision-making. Recent Vision-Language Models (VLMs) have shown promising performance through parameter-efficient fine-tuning; however, most existing approaches rely on coarse visual grounding, typically limited to bounding boxes, which fails to capture the fine-grained spatial structure of surgical objects. In this work, we propose a unified framework that jointly performs pixel-level segmentation and visual question answering within a single framework. Our approach integrates a VLM with a Segment Anything Model (SAM)-based decoder and represents scene elements as object tokens generated by the VLM. These object tokens guide answer prediction and are further projected to the SAM-based decoder to produce segmentation masks. By optimizing the object token embeddings through both segmentation and question answering objectives, the model learns spatially grounded representations that enhance visual reasoning while providing explicit pixel-level grounding. We evaluate the proposed method on the private RAMIE (Robot-Assisted Minimally Invasive Esophagectomy) dataset and the public EndoVis18 dataset, where it consistently outperforms baseline methods for surgical VQA. These results demonstrate that incorporating context-aware object tokens into vision-language models improves fine-grained surgical scene understanding.

手术AI视觉问答分割对象令牌

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。