首个多层级内镜黏膜下剥离术数据集，助力大模型辅助手术操作。
CoPESD: A Multi-Level Surgical Motion Dataset for Training Large Vision-Language Models to Co-Pilot Endoscopic Submucosal Dissection
- 按手术动作层级分解，标注精细动作序列。
- 含35小时视频、17679张图像及88395个动作标注。
- 适合研究智能手术机器人协同与自动化系统。
黏膜下剥离术（ESD）可快速切除大病灶，降低复发率并提升长期生存率。然而其技术难度高，易引发并发症，需依赖经验丰富的外科医生和精准器械。近年来，大视觉语言模型（LVLMs）在机器人决策支持与预测规划方面展现潜力，有助于提升ESD精度并降低风险。但现有针对多层级细粒度ESD运动理解的数据集稀缺且缺乏详细标注。本文提出一种分层动作分解方法，构建首个多层级内镜黏膜下剥离术运动数据集CoPESD，用于训练LVLM作为手术机器人“协作者”。CoPESD涵盖超过35小时的机器人辅助与传统手术视频，包含17,679张图像、32,699个边界框及88,395个多层级动作标注。实验证明，该数据集能有效训练LVLM预测后续手术动作。作为首个多模态ESD运动数据集，CoPESD推动了指令跟随与手术自动化的前沿研究。数据集已开源：https://github.com/gkw0010/CoPESD。
原文摘要 · Abstract (English)
submucosal dissection (ESD) enables rapid resection of large lesions, minimizing recurrence rates and improving long-term overall survival. Despite these advantages, ESD is technically challenging and carries high risks of complications, necessitating skilled surgeons and precise instruments. Recent advancements in Large Visual-Language Models (LVLMs) offer promising decision support and predictive planning capabilities for robotic systems, which can augment the accuracy of ESD and reduce procedural risks. However, existing datasets for multi-level fine-grained ESD surgical motion understanding are scarce and lack detailed annotations. In this paper, we design a hierarchical decomposition of ESD motion granularity and introduce a multi-level surgical motion dataset (CoPESD) for training LVLMs as the robotic \textbf{Co}-\textbf{P}ilot of \textbf{E}ndoscopic \textbf{S}ubmucosal \textbf{D}issection. CoPESD includes 17,679 images with 32,699 bounding boxes and 88,395 multi-level motions, from over 35 hours of ESD videos for both robot-assisted and conventional surgeries. CoPESD enables granular analysis of ESD motions, focusing on the complex task of submucosal dissection. Extensive experiments on the LVLMs demonstrate the effectiveness of CoPESD in training LVLMs to predict following surgical robotic motions. As the first multimodal ESD motion dataset, CoPESD supports advanced research in ESD instruction-following and surgical automation. The dataset is available at \href{https://github.com/gkw0010/CoPESD}{https://github.com/gkw0010/CoPESD.}}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。