arXiv:2608.15336cs.CV2026-08中稿 · BMVC 2026

用提示词替代标注,实现无需定位监督的手术场景图生成。

SAGE-OR: Semi-supervised Adaptive Scene Graph Generation for Operating Rooms

论文配图:SAGE-OR: Semi-supervised Adaptive Scene Graph Generation for Operating Rooms
图 1 · 摘自论文原文
  • 用预计算特征+轻量图变压器,解耦定位与关系推理。
  • 在4D-OR上达76% F1,加手部增强后达86%,接近顶尖水平。
  • 仅需类别和关系标签,适合手术室边缘设备部署。

现有手术场景图生成方法依赖密集多模态监督和专用硬件(如同步RGB-D传感器、标定装置),导致数据集构建成本高,且所有基准测试均局限于仿真环境。本文提出SAGE-OR,一种以特征为中心的半监督框架,将传统‘检测后推理’范式改为解耦的表示-推理范式:定位信息由冻结的基础模型提取,隐式编码于预计算特征中,无需任何定位监督;轻量级图变压器则在缓存特征上执行关系推理。采用通用分割提示的半监督设计,消除定位标注,并通过额外提示驱动实体(如手)实现无监督上下文增强。通用提示提升召回率,精度交由下游注意力推理处理,支持仅通过修改提示快速适配新实体。该设计使图变压器仅1500万参数,训练仅需1.4小时,每帧推理耗时约1毫秒,峰值内存低于2GB,适合手术室边缘设备;特征提取可离线完成,每帧4.27秒。在4D-OR基准上,核心模型达到76% F1,媲美全监督基线,无监督手部增强后提升至86%,距需密集多模态监督的最先进方法仅差4个百分点,为无需标注即可适应新手术场景提供了可行路径。

原文摘要 · Abstract (English)

Current surgical scene graph generation methods depend on dense multi-modal supervision and specialized hardware (synchronized RGB-D sensors, calibration rigs), making dataset construction expensive and restricting all existing benchmarks to simulated environments. We propose SAGE-OR, a feature-centric framework that replaces the traditional detect-then-reason paradigm with a decoupled representation-reasoning paradigm in which localization is derived from frozen foundation models, encoded implicitly in pre-computed features, and used without any localization supervision, while a lightweight graph transformer performs relational reasoning over cached features. We employ a semi-supervised formulation with general-purpose segmentation prompts to eliminate localization supervision while enabling unsupervised context augmentation through additional prompt-driven entities, such as hands, which are absent from annotations. General-purpose prompts are used to induce near-perfect recall, while precision is delegated to downstream attention-based reasoning, enabling simple adaptation to new entities via prompt-level modification. This design enables a lightweight 15M-parameter graph transformer that trains in 1.4 hours and runs relational inference at $\sim$1ms per frame with peak memory under 2GB, suitable for edge hardware used in the operating room; feature extraction runs offline as a separate caching stage (4.27s per frame). On the 4D-OR benchmark, the core model achieves 76% F1, matching the fully supervised 4D-OR baseline while eliminating all localization annotations, and unsupervised hand augmentation raises this to 86%, within 4 points of state-of-the-art (SOTA) methods requiring dense multi-modal supervision, providing a practical pathway for adaptation to new surgical settings without annotation other than relationship and class labels.

场景图生成手术视觉半监督学习边缘部署

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。