arXiv:2602.14445cs.LGcs.AI2026-02被引 2

用振荡器同步机制替代注意力,实现高效稀疏计算。

Selective Synchronization Attention

  • 以振荡器相位锁定原理构建新型注意力,自动实现稀疏性。
  • 无需显式位置编码,自然频率谱统一编码位置与语义。
  • 单次计算完成,比传统方法更快,适合对效率敏感场景。

Transformer架构已成为现代深度学习的基石,但其核心自注意力机制存在二次计算复杂度问题,且缺乏生物神经计算依据。本文提出选择性同步注意力(SSA),用基于耦合振子稳态解的闭式算子取代标准点积自注意力。每个令牌被建模为具有可学习固有频率和相位的振子;令牌对间的同步强度由频率相关耦合和相位锁定条件决定,作为注意力权重。该框架具备三大优势:(i) 由于相位锁定阈值自然产生稀疏性,频率不匹配的令牌自动获得零注意力权重,无需显式掩码;(ii) 通过自然频率谱统一编码位置与语义,无需单独的位置编码;(iii) 单次闭式计算避免迭代微分方程求解,所有组件(耦合、序参量、同步)均源自振荡框架。我们在振荡同步网络(OSN)中实现SSA,作为Transformer模块的即插即用替代品。对同步矩阵的分析显示,即使在初始化阶段也呈现非均匀、头多样化的耦合模式,表明其结构归纳偏置强于随机初始化Transformer所产生的近似均匀注意力。

原文摘要 · Abstract (English)

The Transformer architecture has become the foundation of modern deep learning, yet its core self-attention mechanism suffers from quadratic computational complexity and lacks grounding in biological neural computation. We propose Selective Synchronization Attention (SSA), a novel attention mechanism that replaces the standard dot-product self-attention with a closed-form operator derived from the steady-state solution of the Kuramoto model of coupled oscillators. In SSA, each token is represented as an oscillator characterized by a learnable natural frequency and phase; the synchronization strength between token pairs, determined by a frequency-dependent coupling and phase-locking condition, serves as the attention weight. This formulation provides three key advantages: (i) natural sparsity arising from the phase-locking threshold, whereby tokens with incompatible frequencies automatically receive zero attention weight without explicit masking; (ii) unified positional-semantic encoding through the natural frequency spectrum, eliminating the need for separate positional encodings; and (iii) a single-pass, closed-form computation that avoids iterative ODE integration, with all components (coupling, order parameter, synchronization) derived from the oscillatory framework. We instantiate SSA within the Oscillatory Synchronization Network (OSN), a drop-in replacement for the Transformer block. Analysis of the synchronization matrices reveals non-uniform, head-diverse coupling patterns even at initialization, demonstrating a stronger architectural inductive bias than the approximately uniform attention produced by randomly initialized Transformers.

注意力机制振荡器稀疏计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。