arXiv:2412.12843cs.CVcs.AI2024-12中稿 · IROS 2025被引 3

轻量级脉冲变压器网络,让事件相机高效实现语义分割。

SLTNet: Efficient Event-based Semantic Segmentation with Spike-driven Lightweight Transformer-based Networks

  • 用脉冲驱动卷积块与二值掩码变压器块提取特征。
  • 在DDD17和DSEC上分别提升9.06%和9.39% mIoU,能耗低4.58倍。
  • 适合边缘设备部署,推理速度达114 FPS,开源可复现。

事件相机在自动驾驶与机器人中具有高动态范围、低延迟和低功耗的优势,但现有基于人工神经网络(ANN)的语义分割方法存在计算量大、需图像帧输入、能耗高等问题,难以在资源受限的边缘/移动端应用。为此,本文提出SLTNet,一种基于脉冲驱动的轻量级变压器网络。SLTNet采用高效的脉冲驱动卷积块(SCBs)提取丰富语义特征,同时减少参数量;为增强长程上下文交互,设计新型脉冲驱动变压器块(STBs),引入二值掩码操作。整体采用高效单分支架构,在保持脉冲神经网络(SNN)低功耗特性的同时,实现高性能。在DDD17和DSEC-Semantic数据集上的实验表明,SLTNet相较当前最优的SNN方法,分别提升9.06%和9.39% mIoU,能耗降低4.58倍,推理速度达114 FPS。代码已开源:https://github.com/longxianlei/SLTNet-v1.0。

原文摘要 · Abstract (English)

Event-based semantic segmentation has great potential in autonomous driving and robotics due to the advantages of event cameras, such as high dynamic range, low latency, and low power cost. Unfortunately, current artificial neural network (ANN)-based segmentation methods suffer from high computational demands, the requirements for image frames, and massive energy consumption, limiting their efficiency and application on resource-constrained edge/mobile platforms. To address these problems, we introduce SLTNet, a spike-driven lightweight transformer-based network designed for event-based semantic segmentation. Specifically, SLTNet is built on efficient spike-driven convolution blocks (SCBs) to extract rich semantic features while reducing the model's parameters. Then, to enhance the long-range contextural feature interaction, we propose novel spike-driven transformer blocks (STBs) with binary mask operations. Based on these basic blocks, SLTNet employs a high-efficiency single-branch architecture while maintaining the low energy consumption of the Spiking Neural Network (SNN). Finally, extensive experiments on DDD17 and DSEC-Semantic datasets demonstrate that SLTNet outperforms state-of-the-art (SOTA) SNN-based methods by at most 9.06% and 9.39% mIoU, respectively, with extremely 4.58x lower energy consumption and 114 FPS inference speed. Our code is open-sourced and available at https://github.com/longxianlei/SLTNet-v1.0.

事件相机语义分割脉冲网络轻量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。