无需标注或领域预训练，用图文对齐实现手术阶段自动分割。
Multimodal Optimal Transport for Training-free Temporal Segmentation in Surgical Robotics
- 通过图文对齐的最优传输融合视觉与语义信息，实现无标注分割。
- 在4个数据集上超越零样本基线，最高提升33.7点F1值。
- 适合缺乏标注数据的机器人手术系统实时部署。
自动化识别手术阶段与步骤是机器人辅助手术中术中决策支持、流程自动化和技能评估的基础能力。现有方法依赖大规模标注数据集或数千个带标签视频的领域特定预训练，限制了其在多样机器人平台和临床环境中的实际部署。本文提出TASOT（文本增强的动作分割最优传输）框架，一种无需任务标注或外科领域预训练的注释自由手术时间分割方法。TASOT 在动作分割最优传输（ASOT）基础上，引入直接从输入视频生成的时间对齐文本描述，通过统一的非平衡格罗莫夫-沃瑟斯坦最优传输目标融合视觉与语义线索。视觉特征使用DINOv3提取，视觉-语言模型生成的时序字幕经CLIP编码并对齐至各帧，为传输代价提供互补语义结构。在三个公开外科数据集和四个基准设置下评估，涵盖腹腔镜与机器人手术，结果显著优于最强零样本基线：Cholec80上+18.9 F1，AutoLaparo上+33.7，StrasByPass70上+23.7，BernByPass70上+4.5。结果表明，无需人工训练标注或外科特异性预训练流程，即可实现机器人环境中精细的手术流程理解，为真实世界机器人手术系统提供了有前景的替代方案。
原文摘要 · Abstract (English)
Automated recognition of surgical phases and steps is a fundamental capability for intraoperative decision support, workflow automation, and skill assessment in robotic-assisted surgery. Existing approaches either depend on large-scale annotated surgical datasets or require expensive domain-specific pretraining on thousands of labeled videos, limiting their practical deployability across diverse robotic platforms and clinical environments. In this work, we propose TASOT (Text-Augmented Action Segmentation Optimal Transport), an annotation-free framework for surgical temporal segmentation that requires no task-specific annotations or surgical-domain pretraining. TASOT extends the Action Segmentation Optimal Transport (ASOT) formulation by incorporating temporally aligned textual descriptions generated directly from the input video, fusing visual and semantic cues within a unified unbalanced Gromov-Wasserstein optimal transport objective. Visual representations are extracted using DINOv3, while temporal captions produced by a vision-language model are encoded via CLIP and temporally aligned to individual frames, providing complementary semantic structure to the transport cost. We evaluate TASOT on three public surgical datasets and four benchmark settings spanning laparoscopic and robotic procedures, showing substantial improvements over the strongest zero-shot baselines: +18.9 F1 on Cholec80, +33.7 on AutoLaparo, +23.7 on StrasByPass70, and +4.5 on BernByPass70. These results suggest that fine-grained surgical workflow understanding in robotic settings can be achieved without manual training annotations or surgical-specific pretraining pipelines, offering a promising alternative for real-world robotic surgical systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。