用脉冲神经网络实现低延迟高能效的手术场景分割,适合实时手术辅助系统。
Toward Real-Time Surgical Scene Segmentation via a Spike-Driven Video Transformer with Spike-Informed Pretraining
- 基于脉冲活动引导的掩码预训练,适配稀疏脉冲表示
- 在端口18和自建数据集上达到与顶尖人工神经网络相当的分割精度
- 相比主流大模型提速超20倍,适合资源受限的手术环境
现代手术系统日益依赖智能场景理解以提升术中安全与态势感知,其中手术场景分割在细粒度手术感知中起基础作用。尽管近期人工神经网络(ANN)模型,尤其是大型基础模型,在精度上表现优异,但其高计算与能耗常阻碍在资源受限的术中环境部署。为此,我们探索脉冲神经网络(SNN)这一高效范式。然而,其在手术场景分割中的性能受限于稀疏脉冲表示与标注数据稀缺。因此,我们提出SpikeSurgSeg,首个用于手术场景分割的脉冲驱动视频变换器。它保留了SNN的实时性与低功耗优势,同时在数据稀缺的手术场景中达到与多数ANN模型相当的性能。具体而言,我们引入基于MAE的脉冲感知预训练策略,掩码生成由脉冲发放活动引导,更契合稀疏脉冲表示;并采用分层管状掩码方案,减少信息泄露,促进上下文推理。为增强语义表征,我们设计多谱知识蒸馏,对齐教师模型(ANN)与学生模型(SNN)在频域的特征,有效缓解连续激活模式与脉冲驱动时序表示间的不匹配。基于预训练的SNN编码器,我们进一步设计轻量级脉冲驱动分割头。在EndoVis18和自建SurgBleed数据集上的大量实验表明,SpikeSurgSeg在保持与最先进ANN模型相当的平均交并比(mIoU)的同时,推理延迟降低至少8倍。显著地,其相对于多数基础模型实现超过20倍的加速。
原文摘要 · Abstract (English)
Modern surgical systems increasingly rely on intelligent scene understanding to improve intra-operative safety and situational awareness, with surgical scene segmentation playing a fundamental role in fine-grained surgical perception. Although recent ANN models, especially large foundation models, have achieved impressive accuracy, their high computational and energy demands often hinder deployment in resource-constrained operative environments. To address this challenge, we explore SNN as a highly efficient paradigm. However, its performance in surgical scene segmentation remains constrained by sparse spike representations and limited annotated surgical data. We therefore propose SpikeSurgSeg, the first spike-driven video Transformer for surgical scene segmentation. It preserves the real-time and energy-efficient advantages of SNN, while achieving competitive performance against most ANN models in data-scarce surgical scenarios. Specifically, we introudce a spike-informed pretraining strategy based on MAE, where mask generation is guided by spike firing activity to better align with sparse spike representations, together with a layer-wise tube masking scheme that reduces information leakage and encourages contextual reasoning. To further strengthen semantic representation, we introduce multi-spectral knowledge distillation, which aligns teacher ANN and student SNN features in the frequency domain, where the mismatch between continuous activation patterns and spike-driven temporal representations can be effectively mitigated. Built on the pretrained SNN encoder, we further design a lightweight spike-driven segmentation head. Extensive experiments on EndoVis18 and our in-house SurgBleed dataset show that SpikeSurgSeg achieves mIoU comparable to SOTA ANN models while reducing inference latency by at least 8x. Notably, it delivers over 20x speedup relative to most foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。