arXiv:2606.12059cs.LGcs.NE2026-06

用振荡器同步模拟注意力,低功耗硬件上更高效

Attention by Synchronization in Coupled Oscillator Networks

  • 用相位同步的振荡器网络替代softmax计算
  • 2维振荡器在关键词识别上比softmax高1.00个百分点
  • 适合低功耗物理芯片实现,理论稳定且无需训练失败

本文研究在能量受限的物理硬件上实现Transformer注意力。传统softmax需指数运算和全局归约,能耗高且无自然物理对应。我们发现柯尔莫哥洛夫同步动力学(出现在电、机械、超导等系统中)可自然实现注意力,无需上述操作。提出的固定查询振荡器注意力将计算转化为球面梯度流的平衡过程:查询作为球面上的固定锚点,自由振荡器在柯尔莫哥洛夫-洛赫动力学下演化,最终以余弦相似度编码注意力权重。计算仅依赖平衡过程,无需指数运算;唯一全局操作是读出时的仿射归一化。固定点在几乎所有初始条件下唯一且全局吸引,适用于所有物理实现。实验表明,在最小配置(振荡器维度 $d_{ ext{osc}} = 2$)下,该方法在关键词识别任务上优于softmax(+1.00 pp),在复杂句主谓一致任务上提升5.27 pp,且零训练失败(对比softmax五次中有一次)。在因果语言建模中,随着 $d_{ ext{osc}}$ 增大,性能差距缩小:在WikiText-2上,从 $d_{ ext{osc}}=2$ 时的+11.09 PPL降至 $d_{ ext{osc}}=32$ 时的+2.98 PPL;在TinyStories上,从+2.39降至+0.57。本工作目标非取代软件中的softmax,而是为物理载体提供数学严谨的注意力实现蓝图。

原文摘要 · Abstract (English)

We address transformer attention on energy-constrained physical substrates. Softmax attention requires exponentiation and global reduction, operations with high energy cost on von Neumann hardware and no natural physical analog. We show that Kuramoto synchronization dynamics (which arise in electrical, mechanical, superconducting, and charge-density-wave oscillator arrays, among other physical systems) implement a well-defined attention operation without either. The resulting mechanism, fixed-query oscillator attention, replaces softmax's arithmetic with the equilibration of a gradient flow on the sphere: queries are learned anchors fixed on the sphere, and free oscillators evolve under Kuramoto-Lohe dynamics until they settle at positions encoding attention weights via cosine similarity. Because the computation is equilibration, it requires no exponentiation; the only global operation is an affine normalization at readout. The fixed point is provably unique and globally attractive from almost every initial condition, a guarantee that holds across every physical realization. Empirically, at the minimal hardware configuration (oscillator dimension $d_{\mathrm{osc}}$ = 2), oscillator attention outperforms softmax on keyword spotting (+1.00 pp) and on subject-verb agreement (+5.27 pp on hard sentences, with zero training failures versus one in five for softmax). On causal language modeling, where softmax retains an advantage, oscillator attention closes the gap as $d_{\mathrm{osc}}$ grows: from +11.09 PPL at $d_{\mathrm{osc}}$ = 2 to +2.98 PPL at $d_{\mathrm{osc}}$ = 32 on WikiText-2, and from +2.39 PPL at $d_{\mathrm{osc}}$ = 2 to +0.57 PPL at $d_{\mathrm{osc}}$ = 32 on TinyStories. The main objective of this work is not to replace softmax in software but to provide a mathematically grounded blueprint for accurate attention on physical substrates.

注意力机制振荡器网络低功耗计算物理计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。