用思维链+多智能体分析手术视频,让AI懂手术流程。
SurgRAW: Multi-Agent Workflow with Chain of Thought Reasoning for Robotic Surgical Video Analysis
- 分层推理架构,智能体协同完成任务
- 在14256个问答对上准确率超主流模型14.61%
- 适合医疗AI研究者与临床辅助系统开发者
机器人辅助手术(RAS)是现代外科核心,亟需具备精准场景理解能力的智能系统。现有方法多依赖孤立的任务模型,导致流程碎片化、可解释性差且缺乏统一理解。视觉语言模型(VLMs)虽具零样本推理能力,但存在幻觉、领域差距及任务关联建模弱等问题。为此,我们提出SurgCoTBench——首个聚焦推理的RAS基准,涵盖14256个带帧级标注的QA对,覆盖五大主要手术任务。基于此,我们构建SurgRAW:一种以思维链(CoT)驱动的临床对齐多智能体工作流,实现手术场景的零样本多任务推理。SurgRAW采用分层推理结构,由调度器将任务分解为两个推理流,指派专业智能体生成任务级推理;高层智能体捕捉流程依赖并临床校准输出。我们设计了面板讨论机制,促进任务智能体协同与依赖利用;引入检索增强生成模块,补充手术知识,缓解通用VLM的领域偏差。通过基于手术领域的任务特异性CoT提示,确保推理临床一致,减少幻觉,提升可解释性。大量实验表明,SurgRAW超越主流VLM和代理系统,在监督模型基础上提升14.61%准确率。数据与代码已开源于https://github.com/jinlab-imvr/SurgRAW.git。
原文摘要 · Abstract (English)
Robotic-assisted surgery (RAS) is central to modern surgery, driving the need for intelligent systems with accurate scene understanding. Most existing surgical AI methods rely on isolated, task-specific models, leading to fragmented pipelines with limited interpretability and no unified understanding of RAS scene. Vision-Language Models (VLMs) offer strong zero-shot reasoning, but struggle with hallucinations, domain gaps and weak task-interdependency modeling. To address the lack of unified data for RAS scene understanding, we introduce SurgCoTBench, the first reasoning-focused benchmark in RAS, covering 14256 QA pairs with frame-level annotations across five major surgical tasks. Building on SurgCoTBench, we propose SurgRAW, a clinically aligned Chain-of-Thought (CoT) driven agentic workflow for zero-shot multi-task reasoning in surgery. SurgRAW employs a hierarchical reasoning workflow where an orchestrator divides surgical scene understanding into two reasoning streams and directs specialized agents to generate task-level reasoning, while higher-level agents capture workflow interdependencies or ground output clinically. Specifically, we propose a panel discussion mechanism to ensure task-specific agents collaborate synergistically and leverage on task interdependencies. Similarly, we incorporate a retrieval-augmented generation module to enrich agents with surgical knowledge and alleviate domain gaps in general VLMs. We design task-specific CoT prompts grounded in surgical domain to ensure clinically aligned reasoning, reduce hallucinations and enhance interpretability. Extensive experiments show that SurgRAW surpasses mainstream VLMs and agentic systems and outperforms a supervised model by 14.61% accuracy. Dataset and code is available at https://github.com/jinlab-imvr/SurgRAW.git .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。