面向内窥镜视频的高效基础模型,兼顾实时推理与强表征能力。
EndoMamba: An Efficient Foundation Model for Endoscopic Videos via Hierarchical Pre-training
- 采用双向Mamba块与普通Mamba块结合,实现时空建模与高效推理。
- 在4个下游任务中超越现有模型,且保持实时推理速度。
- 适合需要低延迟、高精度的内窥镜手术辅助系统开发者。
基于内窥镜视频的任务(如视觉导航、手术阶段识别)在微创手术中提供实时辅助至关重要。尽管近期视频基础模型展现出潜力,但其应用受限于计算效率低下和因内窥镜数据有限导致的性能不足。为此,我们提出EndoMamba,一种专为实时推理设计的基础模型,可学习通用时空表示。首先,为缓解计算低效问题,提出优化的EndoMamba骨干网络:借鉴状态空间模型进展,融合双向Mamba块用于帧内空间建模,以及普通Mamba块用于跨时间域的过去到当前推理,实现强大时空建模与在线视频流的高效推理。其次,提出自监督分层预训练方案,利用内窥镜视频并融合通用视频领域知识增强表征学习。具体地,结合掩码重建与辅助监督,通过低层重建捕捉时空结构,高层对齐从预训练通用视频基础模型迁移更广泛知识。在四个下游任务(分类、分割、手术阶段识别、定位)上的大量实验表明,EndoMamba优于现有基础模型与专用方法,同时保持实时推理速度。代码已开源:https://github.com/TianCuteQY/EndoMamba。
原文摘要 · Abstract (English)
Endoscopic video-based tasks, such as visual navigation and surgical phase recognition, play a crucial role in minimally invasive surgeries by providing real-time assistance. While recent video foundation models have shown promise, their applications are hindered by (1) computational inefficiencies and (2) suboptimal performance caused by limited data for pre-training in endoscopy. To address these issues, we present EndoMamba, a foundation model designed for real-time inference while learning generalized spatiotemporal representations. First, to mitigate computational inefficiencies, we propose the EndoMamba backbone, optimized for real-time inference. Inspired by recent advancements in state space models, EndoMamba integrates Bidirectional Mamba blocks for spatial modeling within individual frames and vanilla Mamba blocks for past-to-present reasoning across the temporal domain. This design enables both strong spatiotemporal modeling and efficient inference in online video streams. Second, we propose a self-supervised hierarchical pre-training diagram to enhance EndoMamba's representation learning using endoscopic videos and incorporating general video domain knowledge. Specifically, our approach combines masked reconstruction with auxiliary supervision, leveraging low-level reconstruction to capture spatial-temporal structures and high-level alignment to transfer broader knowledge from a pretrained general-video domain foundation model. Extensive experiments on four downstream tasks--classification, segmentation, surgical phase recognition, and localization--demonstrate that EndoMamba outperforms existing foundation models and task-specific methods while maintaining real-time inference speed. The source code is available at https://github.com/TianCuteQY/EndoMamba.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。