用相位同步机制改进自注意力,让模型计算可解释。
Kuramoto Attention: Synchronizing Self-Attention on the Torus
- 用环面相位振子模拟注意力,通过同步实现值更新。
- 在CodeParrot上500万参数时,平均比特率降低0.012(验证)和0.010(测试)。
- 结果可直接解读为相位状态的自适应同步,适合研究神经机制者。
Transformer 模型被广泛用于认知建模与神经表征,其自注意力机制不仅关乎工程性能,更引发对认知机制的关注。本文引入柯朗莫托注意力层(Kuramoto Attention),将振子同步机制融入自注意力:每个标记携带一组相位振子,隐藏状态位于高维环面;注意力权重构成自适应耦合图,原始相位状态作为值,使值更新恰好对应固定注意力权重下的柯朗莫托耦合方向。Softmax决定哪些振子耦合,值路径则将每个标记推向所选标记的加权圆均值。在enwiki8和CodeParrot上训练,与匹配参数的RoPE和SwiGLU Transformer对比:500万参数下,CodeParrot的平均比特率降低0.012(验证)和0.010(测试);enwiki8上六次实验均低于基准,平均差异在0.01 BPC以内,五次形成紧密低均值簇;100万参数时,在enwiki8上落后约0.02 BPC,CodeParrot落后0.013–0.015 BPC。消融实验与相位诊断揭示同步与几何结构对性能的影响。最终实现可直接解读为相位状态自适应同步的注意力机制。
原文摘要 · Abstract (English)
Transformer models are increasingly used as computational models of cognition and neural representation, so the mechanism implemented by self-attention is of interest beyond engineering performance. A complementary tradition in cognitive science models coordination, binding, and memory through dynamical interactions such as oscillator synchrony; we bring this mechanism into self-attention by introducing the Kuramoto Attention layer, whose value update is a synchronization step. Each token carries a bank of phase oscillators, so its hidden state lives on a high-dimensional torus. The attention weights form an adaptive coupling graph, and using the raw phase states as values makes the value update exactly the Kuramoto coupling direction for fixed attention weights. The softmax selects which oscillators couple, while the value path moves each token toward the attention-weighted circular mean of the tokens it selects. We train Kuramoto Attention on enwiki8 and CodeParrot against parameter-matched RoPE and SwiGLU transformers. At 5M parameters on CodeParrot, it improves on the transformer by both median and mean, with mean gaps of 0.012 validation and 0.010 test bits per byte. At 5M on enwiki8, all six runs have lower validation/test medians than the transformer and all-seed means within 0.01 BPC; five of six also form a tight lower-mean cluster. At 1M, it trails by about 0.02 BPC on enwiki8 and by 0.013-0.015 bits per byte on CodeParrot. Ablations and phase diagnostics show how the layer's synchronization and geometry-motivated components shape model performance. The result is a self-attention mechanism whose learned computation can be read directly as adaptive synchronization on phase states.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。