arXiv:2505.14535cs.LGcs.HC2025-05被引 15

提出动态融合机制,让不同模态的神经脉冲数据更协同地处理。

Spiking Neural Networks with Temporal Attention-Guided Adaptive Fusion for imbalanced Multi-modal Learning

  • 用时序注意力动态调整各模态融合权重,实现时间维度自适应融合。
  • 在三个数据集上达到77.55%~97.5%准确率,显著提升不平衡多模态学习效果。
  • 适合研究神经形态计算与多模态感知的工程师和研究人员。

多模态脉冲神经网络(SNN)在低功耗传感处理中具有巨大潜力,但面临模态不平衡与时间错位的挑战。现有方法存在各模态收敛速度不协调、融合机制静态等问题。本文提出时序注意力引导的自适应融合框架,包含两个创新:1)时序注意力引导的自适应融合(TAAF)模块,在每个时间步动态分配脉冲特征的重要程度,实现时序异构脉冲特征的分层融合;2)基于注意力得分调节各模态学习率的时序自适应平衡融合损失,防止主导模态垄断优化过程。该框架在CREMA-D、AVE和EAD数据集上分别取得77.55%、70.65%和97.5%的准确率,兼具性能与能效。系统通过可学习的时间扭曲操作解决时间错位问题,实现比基线SNN更快的模态收敛协调。本工作为神经形态系统中的时序一致多模态学习建立新范式,弥合生物感知与高效机器智能的差距。

原文摘要 · Abstract (English)

Multimodal spiking neural networks (SNNs) hold significant potential for energy-efficient sensory processing but face critical challenges in modality imbalance and temporal misalignment. Current approaches suffer from uncoordinated convergence speeds across modalities and static fusion mechanisms that ignore time-varying cross-modal interactions. We propose the temporal attention-guided adaptive fusion framework for multimodal SNNs with two synergistic innovations: 1) The Temporal Attention-guided Adaptive Fusion (TAAF) module that dynamically assigns importance scores to fused spiking features at each timestep, enabling hierarchical integration of temporally heterogeneous spike-based features; 2) The temporal adaptive balanced fusion loss that modulates learning rates per modality based on the above attention scores, preventing dominant modalities from monopolizing optimization. The proposed framework implements adaptive fusion, especially in the temporal dimension, and alleviates the modality imbalance during multimodal learning, mimicking cortical multisensory integration principles. Evaluations on CREMA-D, AVE, and EAD datasets demonstrate state-of-the-art performance (77.55\%, 70.65\% and 97.5\%accuracy, respectively) with energy efficiency. The system resolves temporal misalignment through learnable time-warping operations and faster modality convergence coordination than baseline SNNs. This work establishes a new paradigm for temporally coherent multimodal learning in neuromorphic systems, bridging the gap between biological sensory processing and efficient machine intelligence.

脉冲神经网络多模态学习时序融合神经形态计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。