用少量标注数据实现手术流程分析,靠文本驱动模型适配新任务。
Text-driven Adaptation of Foundation Models for Few-shot Surgical Workflow Analysis
- 通过文本对齐图像嵌入,解决图像与文本模态差异。
- 仅用文本训练解码器,无需成对图文数据,支持图像任务。
- 在少样本下跨任务泛化好,适合医疗影像小数据场景。
手术流程分析对提升手术效率与安全性至关重要,但以往研究高度依赖大规模标注数据,存在成本高、可扩展性差及对专家标注的依赖问题。为此,我们提出Surg-FTDA(少样本文本驱动适配),可在极少量配对图像-标签数据下完成多种手术流程分析任务。方法包含两个核心组件:首先,基于少样本选择的模态对齐,从少量图像中选出样本并将其嵌入与下游任务的文本嵌入对齐,弥合模态差距;其次,文本驱动适配仅利用文本数据训练解码器,无需成对图文数据,再将该解码器应用于对齐后的图像嵌入,实现无需显式图文对的图像相关任务。我们在生成任务(图像描述生成)和判别任务(三元组识别、阶段识别)上评估该方法,结果表明Surg-FTDA优于基线模型,并在下游任务间具有良好泛化能力。结论:提出一种文本驱动的适配方法,有效缓解模态差距,在低依赖大规模标注数据的前提下,实现多任务手术流程分析。代码与数据集将发布于https://github.com/CAMMA-public/Surg-FTDA。
原文摘要 · Abstract (English)
Purpose: Surgical workflow analysis is crucial for improving surgical efficiency and safety. However, previous studies rely heavily on large-scale annotated datasets, posing challenges in cost, scalability, and reliance on expert annotations. To address this, we propose Surg-FTDA (Few-shot Text-driven Adaptation), designed to handle various surgical workflow analysis tasks with minimal paired image-label data. Methods: Our approach has two key components. First, Few-shot selection-based modality alignment selects a small subset of images and aligns their embeddings with text embeddings from the downstream task, bridging the modality gap. Second, Text-driven adaptation leverages only text data to train a decoder, eliminating the need for paired image-text data. This decoder is then applied to aligned image embeddings, enabling image-related tasks without explicit image-text pairs. Results: We evaluate our approach to generative tasks (image captioning) and discriminative tasks (triplet recognition and phase recognition). Results show that Surg-FTDA outperforms baselines and generalizes well across downstream tasks. Conclusion: We propose a text-driven adaptation approach that mitigates the modality gap and handles multiple downstream tasks in surgical workflow analysis, with minimal reliance on large annotated datasets. The code and dataset will be released in https://github.com/CAMMA-public/Surg-FTDA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。