arXiv:2606.18694cs.LGcond-mat.dis-nn2026-06被引 1

用同步受阻机制构建注意力模型,性能超越传统变压器。

Attention as Frustrated Synchronization

  • 以相位同步受阻建模注意力,通过谐波与延迟项实现动态计算。
  • 在字符级文本和代码任务中,损失低于调优后的RoPE-SwiGLU Transformer。
  • 无需多层感知机,仅靠集体模式耦合也能逼近变压器性能。

一个完全同步的振荡器网络无法进行额外计算,因此基于同步的注意力架构必须在结构化的非一致状态中寻找计算能力。我们提出受阻同步网络(FSN),其令牌状态为环面上的相位,整个价值路径仅由一个学习得到的复数耦合核(涵盖谐波与单步延迟)构成。核中的每个分量均对应同步文献中的‘受阻’概念:静态的Kuramoto-Sakaguchi相位差、符号谐波代表排斥性Daido分量,而延迟项将每个令牌与它所关注令牌的后继耦合,其代数形式与带有数据自身转移作为受阻角的Kuramoto-Sakaguchi耦合一致,从而将下一个词预测转化为由数据本身造成的同步受阻。在字符级文本和代码任务中,以一百万参数和相同训练预算对比,FSN在每一轮迭代中验证损失均低于调优后的RoPE-SwiGLU Transformer;即使基线训练至收敛,每次三十轮的enwik8种子结果仍低于变换器收敛后的50轮损失1.611,而FSN的50轮运行收敛至1.5953 ± 0.0014。将每个前馈模块替换为对学习到的集体模式的平均场耦合,移除所有多层感知机后,模型仍能追踪变压器表现。在自然文本上,无受阻的基础层在所有复制深度下均落后于收敛的变压器,尤其在长距离复制事件中表现最差;而引入受阻后,所有四层及以上的深度均逆转劣势。主要对比在百万参数规模完成,参数量阶梯扩展至四百万,优势持续存在,其余分支仍在推进中。

原文摘要 · Abstract (English)

A network of oscillators that synchronizes perfectly computes nothing further, so an attention architecture built from synchronization must locate its computation in structured departures from agreement. We introduce the Frustrated Synchronization Network (FSN), whose token states are phases on a torus and whose entire value pathway is one learned complex coupling kernel over harmonics and a one-step delay. Each component of the kernel is a frustration in the sense of the synchronization literature. The complex phases are static Kuramoto-Sakaguchi frustration angles, the signed harmonics are repulsive Daido components, and the delay term, which couples each token to the successors of the tokens it attends to, is algebraically identical to Kuramoto-Sakaguchi coupling whose frustration angle is the data's own transition, so next-token prediction is implemented as synchronization frustrated by the data. At matched one-million-parameter and training budgets on character-level text and code, the FSN's validation loss is below a tuned RoPE-SwiGLU transformer's at every epoch measured, and the comparison survives training the baseline to convergence: every thirty-epoch enwik8 seed finishes below the transformer's converged fifty-epoch loss of 1.611, and the FSN's completed fifty-epoch runs converge to 1.5953 +/- 0.0014. A variant with every feed-forward block replaced by mean-field coupling to learned collective modes, leaving no multilayer perceptron in the stack, tracks the transformer. On natural text the unfrustrated base layer falls behind the converged transformer at every copy depth, worst on long-range copy events; the kernel reverses the deficit at every depth of four and beyond. Headline comparisons are at the one-million-parameter scale; a scale ladder is complete through four million parameters with the advantage persisting, and remaining arms are marked as in progress.

注意力机制同步系统生成模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。