让查询和键共同演化，提升语言模型性能与训练稳定性。
Coupled Query-Key Dynamics for Attention
- 通过共享动态机制同步更新查询和键，而非独立处理。
- 在WikiText-103上将困惑度降至22.55–22.62，比标准注意力降低6.6%–6.9%。
- 仅增加0.11%参数，且单步耦合已足够，适合追求高效训练的场景。
标准缩放点积注意力使用静态、独立的输入投影计算得分。本文表明，在打分前通过共享学习的动态过程协同演化查询和键——即「耦合QK动态」——可显著提升语言建模的困惑度与训练稳定性。在6000万参数的WikiText-103上，耦合动态实现22.55–22.62的困惑度，相较标准注意力（24.22）降低6.6%–6.9%，且仅增加0.11%参数（跨两种实例共享）。结构消融实验表明，耦合是关键因素：当同时耦合查询与键时，辛几何（哈密顿）与非辛（欧拉）积分器表现一致；而容量相当的解耦MLP基线仅达23.81，且种子方差高8倍。积分步数（1–7）影响微弱，单步已足够。计算量匹配对比显示，耦合是一种样本效率机制：标准注意力需延长2.4倍训练时间（相同墙钟时间）才能达到同等效果，但需2.4倍更多文本 token。该优势在1.5亿参数下仍保持（降低6.7%），但在3.5亿参数下减弱至1.0%；此时微分注意力（18.93）优于耦合动态（19.35）。收益具有语料依赖性：在领域一致文本（WikiText-103降低6.6%，PubMed降低4.5%）中有效，但在异构网络文本中恶化（升高10.3%），对GLUE无益。本文揭示了耦合有效与无效的条件，提供实用指导。
原文摘要 · Abstract (English)
Standard scaled dot-product attention computes scores from static, independent projections of the input. We show that evolving queries and keys \emph{jointly} through shared learned dynamics before scoring - which we call \textbf{coupled QK dynamics} - improves language modeling perplexity and training stability. On WikiText-103 at 60M parameters, coupled dynamics achieves 22.55--22.62 perplexity vs.\ 24.22 for standard attention ($-$6.6--6.9\%), with only 0.11\% additional parameters (shared across both instantiations). A structural ablation isolates coupling as the active ingredient: a symplectic (Hamiltonian) and a non-symplectic (Euler) integrator perform identically when both couple Q and K, while an uncoupled MLP baseline of matched capacity reaches only 23.81 with 8$\times$ higher seed variance. The integration step count (1--7) is similarly irrelevant - a single coupled step suffices. A compute-matched comparison reveals that coupling is a \emph{sample-efficiency} mechanism: standard attention trained for 2.4$\times$ longer (matching wall-clock) reaches the same perplexity, but requires 2.4$\times$ more tokens. The advantage scales to 150M ($-$6.7\%) but narrows at 350M ($-$1.0\%), where Differential Attention (18.93) overtakes coupled dynamics (19.35). The benefit is corpus-dependent: coupling helps on domain-coherent text (WikiText-103 $-$6.6\%, PubMed $-$4.5\%) but degrades on heterogeneous web text ($+$10.3\%) and shows no benefit on GLUE. We characterize when coupling helps and when it does not, providing practical guidelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。