用脉冲神经网络实现低功耗立体图像修复,效果媲美传统模型。
SNNSIR: A Simple Spiking Neural Network for Stereo Image Restoration
- 全脉冲驱动架构,避免浮点运算,适配硬件部署。
- 在雨痕去除、低光增强等任务中性能接近主流模型,计算量大幅降低。
- 适合需要实时低功耗运行的嵌入式视觉系统开发者。
脉冲神经网络(SNN)以离散二进制脉冲传递信息,具备高计算效率和低能耗优势,适用于立体图像修复等计算密集型任务。本文提出SNNSIR,一种专为脉冲驱动范式设计的简单高效立体图像修复SNN。不同于依赖浮点运算(如矩阵除法、指数运算)的混合SNN-ANN模型,本方法采用全脉冲驱动架构,实现低功耗与硬件友好计算。为克服二进制脉冲神经元表达能力不足的问题,提出轻量级脉冲残差基本模块(SRBB),通过脉冲兼容的残差学习增强信息流动;引入脉冲立体卷积调制模块(SSCM),通过逐元素相乘实现简化非线性,并利用跨视图感知机制突出噪声敏感区域;进一步设计脉冲立体交叉注意力模块(SSCA),在脉冲兼容框架内实现视图间的高效双向特征交互。在雨痕去除、雨滴去除、低光增强和超分辨率等多种立体图像修复任务上进行广泛实验,结果表明:该模型在保持竞争力恢复性能的同时,显著降低计算开销,展现出实现实时、低功耗立体视觉应用的巨大潜力。代码将在论文接收后公开。
原文摘要 · Abstract (English)
Spiking Neural Networks (SNNs), characterized by discrete binary activations, offer high computational efficiency and low energy consumption, making them well-suited for computation-intensive tasks such as stereo image restoration. In this work, we propose SNNSIR, a simple yet effective Spiking Neural Network for Stereo Image Restoration, specifically designed under the spike-driven paradigm where neurons transmit information through sparse, event-based binary spikes. In contrast to existing hybrid SNN-ANN models that still rely on operations such as floating-point matrix division or exponentiation, which are incompatible with the binary and event-driven nature of SNNs, our proposed SNNSIR adopts a fully spike-driven architecture to achieve low-power and hardware-friendly computation. To address the expressiveness limitations of binary spiking neurons, we first introduce a lightweight Spike Residual Basic Block (SRBB) to enhance information flow via spike-compatible residual learning. Building on this, the Spike Stereo Convolutional Modulation (SSCM) module introduces simplified nonlinearity through element-wise multiplication and highlights noise-sensitive regions via cross-view-aware modulation. Complementing this, the Spike Stereo Cross-Attention (SSCA) module further improves stereo correspondence by enabling efficient bidirectional feature interaction across views within a spike-compatible framework. Extensive experiments on diverse stereo image restoration tasks, including rain streak removal, raindrop removal, low-light enhancement, and super-resolution demonstrate that our model achieves competitive restoration performance while significantly reducing computational overhead. These results highlight the potential for real-time, low-power stereo vision applications. The code will be available after the article is accepted.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。