arXiv:2605.00662cs.NEcs.LG2026-05

揭示脉冲网络与Transformer共享核心序列处理机制

Spiking Sequence Machines and Transformers

  • 发现脉冲内存与Transformer均实现五种相同功能操作
  • 证明正弦位置编码本质是距离可区分性而非波形形式
  • 适合研究神经计算与模型架构的跨领域学者

序列学习本质上是基于时间索引表示空间中的相似性检索,这是所有序列模型的约束,而非特定架构的属性。我们表明,2007年的脉冲稀疏分布式记忆序列机与2017年的Transformer独立实现了相同的五种功能操作(编码、上下文保持、关联检索、存储和解码),且两者均以余弦相似性作为共享的检索基础。我们形式化提出相位-延迟同构,证明正弦位置相位与尖峰时序线性相关,并证明点积注意力在该映射下仅受全局尺度因子影响(引理1)。实验证明,频率压缩的位置编码在位置敏感的复制任务中无法收敛,而学习得到的基于秩的嵌入表现与正弦编码相当甚至更优,表明位置表征的关键特性是在点积相似性下的距离可区分性,而非正弦形式。时间、相位与秩是同一计算原语的三种实现方式,其有序索引结构在相似性检索中得以保留。

原文摘要 · Abstract (English)

Sequence learning reduces to similarity-based retrieval over a temporally indexed representation space, a constraint on any sequence model, not a property of a specific architecture. We show that a spiking Sparse Distributed Memory sequence machine (2007) and the transformer (2017) independently instantiate the same five functional operations (encoding, context maintenance, associative retrieval, storage, and decoding), with cosine similarity as the shared retrieval primitive in both. We formalise a Phase-Latency Isomorphism showing that sinusoidal positional phase and spike timing are linearly related, and prove that dot product attention is invariant to this mapping up to a global scale factor on the positional component (Lemma 1). Empirically, frequency-compressed positional encoding fails to converge on a positionally demanding copy task, while a learned rank-based embedding matches or exceeds sinusoidal encoding, indicating that the critical property for positional representation is distance discriminability under dot-product similarity, not sinusoidal form. Time, phase, and rank are three instantiations of the same computational primitive, an ordered index whose structure survives similarity-based retrieval.

脉冲神经网络Transformer位置编码序列建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。