提出脑启发的脉冲网络，实现高效图像文本检索。
Brain-Inspired Multimodal Spiking Neural Network for Image-Text Retrieval
- 在脉冲层面融合多模态特征，提升表示能力
- 仅用2个时间步达顶尖检索精度，能耗极低
- 适合低功耗实时多模态应用，如边缘设备
脉冲神经网络（SNN）在单模态视觉与文本任务中展现出强劲潜力，但构建直接训练、低功耗且高性能的多模态SNN用于图像-文本检索（ITR）仍极具挑战。现有基于人工神经网络（ANN）的方法常依赖更深更复杂的架构以获取丰富单模态语义，却忽视了跨模态交互、检索延迟与能效。为此，我们首次提出脑启发的跨模态脉冲融合网络（CMSF），并应用于ITR。其脉冲融合机制在脉冲层整合单模态特征，生成增强的多模态表示，作为软监督信号优化单模态脉冲嵌入，有效缓解语义损失。尽管仅需两个时间步，CMSF在检索精度上超越现有顶尖的ANN方法，同时保持极低能耗与高速度。本工作标志着多模态SNN的重要进展，提供了一个融合时序动态与跨模态对齐的脑启发框架，为未来基于脉冲的多模态研究提供新思路。代码已开源：https://github.com/zxt6174/CMSF。
原文摘要 · Abstract (English)
Spiking neural networks (SNNs) have recently shown strong potential in unimodal visual and textual tasks, yet building a directly trained, low-energy, and high-performance SNN for multimodal applications such as image-text retrieval (ITR) remains highly challenging. Existing artificial neural network (ANN)-based methods often pursue richer unimodal semantics using deeper and more complex architectures, while overlooking cross-modal interaction, retrieval latency, and energy efficiency. To address these limitations, we present a brain-inspired Cross-Modal Spike Fusion network (CMSF) and apply it to ITR for the first time. The proposed spike fusion mechanism integrates unimodal features at the spike level, generating enhanced multimodal representations that act as soft supervisory signals to refine unimodal spike embeddings, effectively mitigating semantic loss within CMSF. Despite requiring only two time steps, CMSF achieves top-tier retrieval accuracy, surpassing state-of-the-art ANN counterparts while maintaining exceptionally low energy consumption and high retrieval speed. This work marks a significant step toward multimodal SNNs, offering a brain-inspired framework that unifies temporal dynamics with cross-modal alignment and provides new insights for future spiking-based multimodal research. The code is available at https://github.com/zxt6174/CMSF.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。