arXiv:2502.13883cs.CV2025-02被引 3

无需标定的多视角手术动作识别预训练框架,提升复杂手术环境中的动作识别效果。

Multi-view Video-Pose Pretraining for Operating Room Surgical Activity Recognition

  • 采用双编码器结构,对齐多视角视觉与2D姿态嵌入。
  • 通过离散化姿态坐标和掩码预测策略增强跨模态表示学习。
  • 在多视角与单视角设置下均表现优异,适合真实手术场景应用。

复杂手术室中理解手术流程需要深入把握医护人员与环境的交互关系。手术动作识别(SAR)是从多视角摄像头记录中检测动作或阶段的关键计算机视觉任务。现有SAR模型常忽略精细的医护动作及多视角知识,或依赖标定的多视角相机设置和高级点云处理以获得更好结果。本文提出一种新型无标定多视角多模态预训练框架——PreViPS,通过对齐不同视角的2D姿态与视觉嵌入实现跨视图一致性。模型采用类CLIP双编码器架构:一个编码器处理视觉特征,另一个编码人体姿态嵌入。为处理连续的2D人体姿态坐标,我们引入分词离散表示,将连续姿态转为离散姿态嵌入,从而高效融入双编码器框架。通过在嵌入空间中设计跨模态与同模态几何约束,并结合掩码姿态标记预测策略,强化表征学习。大量实验与消融研究验证了其优于强基线的效果;在两个独立的手术室数据集上的数据效率实验进一步凸显该方法的有效性。结果表明,该方法在多视角与单视角设置下均能有效提升手术动作识别性能,展现出在复杂手术环境中的实际应用潜力。代码将开源于:https://github.com/CAMMA-public/PreViPS。

原文摘要 · Abstract (English)

Understanding the workflow of surgical procedures in complex operating rooms requires a deep understanding of the interactions between clinicians and their environment. Surgical activity recognition (SAR) is a key computer vision task that detects activities or phases from multi-view camera recordings. Existing SAR models often fail to account for fine-grained clinician movements and multi-view knowledge, or they require calibrated multi-view camera setups and advanced point-cloud processing to obtain better results. In this work, we propose a novel calibration-free multi-view multi-modal pretraining framework called Multiview Pretraining for Video-Pose Surgical Activity Recognition PreViPS, which aligns 2D pose and vision embeddings across camera views. Our model follows CLIP-style dual-encoder architecture: one encoder processes visual features, while the other encodes human pose embeddings. To handle the continuous 2D human pose coordinates, we introduce a tokenized discrete representation to convert the continuous 2D pose coordinates into discrete pose embeddings, thereby enabling efficient integration within the dual-encoder framework. To bridge the gap between these two modalities, we propose several pretraining objectives using cross- and in-modality geometric constraints within the embedding space and incorporating masked pose token prediction strategy to enhance representation learning. Extensive experiments and ablation studies demonstrate improvements over the strong baselines, while data-efficiency experiments on two distinct operating room datasets further highlight the effectiveness of our approach. We highlight the benefits of our approach for surgical activity recognition in both multi-view and single-view settings, showcasing its practical applicability in complex surgical environments. Code will be made available at: https://github.com/CAMMA-public/PreViPS.

手术识别多视角姿态预训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。