用自然语言统一训练多个手术任务，减少标注量23%
Adaptation of Multi-modal Representation Models for Multi-task Surgical Computer Vision
- 基于CLIP的多模态模型，通过自然语言指令统一处理多个手术任务
- 在多任务数据上实现与专用模型相当的性能，且容忍不完整标注
- 首次将单正例多标签学习用于手术多任务，适合临床标注资源有限场景
手术AI常需同时完成多个任务，如手术阶段识别或腹腔镜胆囊切除术中的关键视野评估。传统模型逐任务构建，缺乏灵活性。为此，我们提出MML-SurgAdapt，一个基于视觉-语言模型（如CLIP）的统一多任务框架，通过自然语言监督处理多样手术任务。多任务学习中常存在部分标注问题，我们采用单正例多标签（SPML）学习，仅需每实例一个正标签即可训练，显著降低标注负担。该框架首次将SPML扩展至整合单个手术过程中的多任务数据，实现有效学习。在Cholec80、Endoscapes2023和CholecT50组合数据集上验证，使用自定义提示词，结果表明模型性能接近专用基准，且对噪声标注具有鲁棒性。相比现有SPML框架，本方法表现更优，标注量减少23%，显著提升标注效率。此工作为手术计算机视觉中的多任务学习提供了一种新颖且通用的解决方案。
原文摘要 · Abstract (English)
Surgical AI often involves multiple tasks within a single procedure, like phase recognition or assessing the Critical View of Safety in laparoscopic cholecystectomy. Traditional models, built for one task at a time, lack flexibility, requiring a separate model for each. To address this, we introduce MML-SurgAdapt, a unified multi-task framework with Vision-Language Models (VLMs), specifically CLIP, to handle diverse surgical tasks through natural language supervision. A key challenge in multi-task learning is the presence of partial annotations when integrating different tasks. To overcome this, we employ Single Positive Multi-Label (SPML) learning, which traditionally reduces annotation burden by training models with only one positive label per instance. Our framework extends this approach to integrate data from multiple surgical tasks within a single procedure, enabling effective learning despite incomplete or noisy annotations. We demonstrate the effectiveness of our model on a combined dataset consisting of Cholec80, Endoscapes2023, and CholecT50, utilizing custom prompts. Extensive evaluation shows that MML-SurgAdapt performs comparably to task-specific benchmarks, with the added advantage of handling noisy annotations. It also outperforms the existing SPML frameworks for the task. By reducing the required labels by 23%, our approach proposes a more scalable and efficient labeling process, significantly easing the annotation burden on clinicians. To our knowledge, this is the first application of SPML to integrate data from multiple surgical tasks, presenting a novel and generalizable solution for multi-task learning in surgical computer vision. Implementation is available at: https://github.com/CAMMA-public/MML-SurgAdapt
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。