arXiv:2409.06217cs.CV2024-09被引 5

通过双流机制自适应捕捉手术阶段上下文,提升在线识别准确率。

DACAT: Dual-stream Adaptive Clip-aware Time Modeling for Robust Online Surgical Phase Recognition

  • 双流设计:一流缓存历史帧特征,一流实时提取当前帧特征。
  • 引入最大片段响应读出模块,动态选择最相关历史片段。
  • 在三个公开数据集上提升显著,最高达4.6%的杰卡德分数改进。

手术阶段识别在腹腔镜手术中至关重要,支持风险预测等临床应用。现有方法通常以单帧嵌入为时间建模基础,但对当前观测过于敏感,导致同一阶段内预测不连贯且错误频发。本文提出DACAT,一种新型双流模型,通过自适应学习片段感知的上下文信息来增强时序关系建模。一个流预训练帧编码器并缓存所有历史帧特征;另一流微调新帧编码器以提取当前帧特征。引入最大片段响应读出(Max-R)模块,利用当前帧特征自适应地从特征缓存中获取最相关的过去片段。将当前帧与所选片段的上下文特征通过交叉注意力融合,进一步用于增强时间建模,实现精准的在线手术阶段识别。在Cholec80、M2CAI16和AutoLaparo三个公开数据集上的基准测试结果表明,所提DACAT优于现有最先进方法,杰卡德得分分别提升至少4.5%、4.6%和2.7%。代码与模型已开源。

原文摘要 · Abstract (English)

Surgical phase recognition has become a crucial requirement in laparoscopic surgery, enabling various clinical applications like surgical risk forecasting. Current methods typically identify the surgical phase using individual frame-wise embeddings as the fundamental unit for time modeling. However, this approach is overly sensitive to current observations, often resulting in discontinuous and erroneous predictions within a complete surgical phase. In this paper, we propose DACAT, a novel dual-stream model that adaptively learns clip-aware context information to enhance the temporal relationship. In one stream, DACAT pretrains a frame encoder, caching all historical frame-wise features. In the other stream, DACAT fine-tunes a new frame encoder to extract the frame-wise feature at the current moment. Additionally, a max clip-response read-out (Max-R) module is introduced to bridge the two streams by using the current frame-wise feature to adaptively fetch the most relevant past clip from the feature cache. The clip-aware context feature is then encoded via cross-attention between the current frame and its fetched adaptive clip, and further utilized to enhance the time modeling for accurate online surgical phase recognition. The benchmark results on three public datasets, i.e., Cholec80, M2CAI16, and AutoLaparo, demonstrate the superiority of our proposed DACAT over existing state-of-the-art methods, with improvements in Jaccard scores of at least 4.5%, 4.6%, and 2.7%, respectively. Our code and models have been released at https://github.com/kk42yy/DACAT.

手术识别时序建模双流网络视频理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。