无需训练,用扩散模型提取特征实现手术视频中物体的精准时序追踪。
Training-free Temporal Object Tracking in Surgical Videos
- 利用预训练扩散模型的图像定位能力提取特征,无需微调或标注。
- 在CholeSeg8K数据集上达到79.19%像素分类准确率,56.20%平均交并比。
- 适合无标注数据、追求低成本部署的医疗视频分析场景。
本文提出一种针对腹腔镜胆囊切除术(LC)手术视频的在线物体追踪方法,旨在定位并追踪关键解剖结构与器械。针对现有数据集标注成本高、标签不一致的问题,本方法利用预训练文本到图像扩散模型的内在定位能力,从手术帧中提取代表性特征,无需任何训练或微调。追踪框架通过受查询-键-值注意力启发的亲和矩阵实现跨帧交互,确保追踪过程的时间连续性。初步研究表明,扩散模型特征在不同解码器层级和时间帧间具有优异的物体定位能力与语义一致性。大规模实验验证了该方法的有效性,在公开的CholeSeg8K数据集上,实现了79.19%的像素级分类准确率、56.20%的平均交并比(Jaccard Score)以及79.48%的平均F-Score,显著优于现有方法。本工作不仅拓展了文本到图像扩散模型的新应用,也为微创手术视频分析提供了高效、低成本的时序追踪解决方案。
原文摘要 · Abstract (English)
Purpose: In this paper, we present a novel approach for online object tracking in laparoscopic cholecystectomy (LC) surgical videos, targeting localisation and tracking of critical anatomical structures and instruments. Our method addresses the challenges of costly pixel-level annotations and label inconsistencies inherent in existing datasets. Methods: Leveraging the inherent object localisation capabilities of pre-trained text-to-image diffusion models, we extract representative features from surgical frames without any training or fine-tuning. Our tracking framework uses these features, along with cross-frame interactions via an affinity matrix inspired by query-key-value attention, to ensure temporal continuity in the tracking process. Results: Through a pilot study, we first demonstrate that diffusion features exhibit superior object localisation and consistent semantics across different decoder levels and temporal frames. Later, we perform extensive experiments to validate the effectiveness of our approach, showcasing its superiority over competitors for the task of temporal object tracking. Specifically, we achieve a per-pixel classification accuracy of 79.19%, mean Jaccard Score of 56.20%, and mean F-Score of 79.48% on the publicly available CholeSeg8K dataset. Conclusion: Our work not only introduces a novel application of text-to-image diffusion models but also contributes to advancing the field of surgical video analysis, offering a promising avenue for accurate and cost-effective temporal object tracking in minimally invasive surgery videos.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。