arXiv:2607.19879cs.CV2026-07

用膜电位融合红外与可见光图像,实现高效低耗的跨模态信息整合。

Current Injection Spiking Neural Network for Infrared and Visible Image Fusion

论文配图:Current Injection Spiking Neural Network for Infrared and Visible Image Fusion
图 1 · 摘自论文原文
  • 在膜电位层面注入电流,实现双模态特征融合前的协同计算
  • 相比同类ANN模型,推理能耗降低约10倍,性能相当
  • 适合追求能效比的实时视觉系统部署

红外与可见光图像融合(IVIF)旨在将两种模态的互补信息整合到一幅图像中,以呈现更丰富的场景内容。现有方法多基于人工神经网络(ANN),需对所有激活值进行密集计算,而脉冲神经网络(SNN)通过稀疏二进制脉冲通信,仅在有脉冲时才计算,具备更高的能效潜力。然而,直接将SNN应用于IVIF存在根本矛盾:跨模态融合依赖于两模态的精细响应,而二进制脉冲可能丢失低于阈值的互补信息。膜电位在放电前保留这些亚阈值响应,使两模态可在该阶段联合塑造输出。基于此,我们提出CIS-Fuse,一种在膜电位层面直接进行跨模态融合的脉冲网络。其核心是电流注入脉冲(CIS)操作器,将一模态作为受控辅助电流注入另一模态的驱动神经元,使两模态在脉冲产生前完成集成,并采用每通道可学习的注入强度自适应调节调制幅度。在此基础上,构建双向跨模态融合(BCMF)模块,并部署于具有非对称堆叠深度的双分支架构中,使两分支形成明确功能分工。在四个IVIF基准数据集及下游检测与分割任务上的大量实验表明,CIS-Fuse在融合质量上达到主流ANN方法水平,同时继承了脉冲计算的能效优势,推理能耗较同规模的ANN基线DCEvo降低约一个数量级。代码将在发表后公开。

原文摘要 · Abstract (English)

Infrared and visible image fusion (IVIF) integrates the complementary information of two modalities into a single image with richer scene content. While existing methods are largely built on artificial neural networks (ANNs), which densely compute over all activations, spiking neural networks (SNNs) communicate through sparse binary spikes and compute only where and when a spike occurs, offering a route to more energy-efficient fusion. However, directly applying SNNs to IVIF creates a fundamental tension: cross-modal fusion relies on fine-grained responses from both modalities, whereas binary spikes can discard complementary cues that remain below the firing threshold. The membrane potential retains these subthreshold responses before firing, letting both modalities jointly shape the output when integrated at this stage. Building on this, we propose CIS-Fuse, a spiking network that performs cross-modal fusion directly at the membrane-potential level. At its core is the current injection spiking (CIS) operator, which injects one modality as a gated auxiliary current into the driving neuron of the other, so the two integrate before spike firing, with a per-channel learnable injection strength that adaptively regulates the modulation magnitude. Building on CIS, we construct a bidirectional cross-modal fusion (BCMF) module and deploy it on a dual-branch architecture with asymmetric stacking depths, where the two branches develop a clear functional specialization. Extensive experiments on four IVIF benchmarks and on downstream detection and segmentation show that CIS-Fuse achieves fusion quality on par with state-of-the-art ANN-based methods while inheriting the energy efficiency of spike-based computation, with roughly an order of magnitude lower inference energy than the similarly-sized ANN-based DCEvo. Code will be released upon publication.

图像融合脉冲神经网络能效优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。