SLAY用球面几何感知的Yat核实现线性时间注意力，逼近softmax性能。
SLAY: Geometry-Aware Spherical Linearized Attention with Yat-Kernel
- 将查询与键约束在单位球面，仅依赖角度对齐计算注意力
- 实现O(L)线性复杂度，性能接近标准softmax注意力
- 适合需要高效长序列建模的Transformer应用
我们提出一类基于简化且计算高效的E-Product（即Yat核）的新线性时间注意力机制。所提方法SLAY（球面线性化注意力带Yat核）将查询和键限制在单位球面上，使注意力仅依赖于角度对齐，灵感源自物理中的平方反比作用。利用Bernstein定理，我们将球面Yat核表示为多项式-指数乘积核的非负混合，并推导出严格正的随机特征近似，实现线性时间复杂度O(L)。我们证明了该核在球面上的正定性与有界性，并表明估计器能生成定义良好且非负的注意力分数。实验表明，SLAY性能几乎与标准softmax注意力无异，同时保持线性时间与内存开销，且持续优于先前的线性注意力机制如Performers和Cosformers。据我们所知，SLAY是迄今为止最接近softmax注意力的线性时间近似，可在不牺牲性能的前提下实现可扩展的Transformer模型。
原文摘要 · Abstract (English)
We propose a new class of linear-time attention mechanisms based on a relaxed and computationally efficient formulation of the recently introduced E-Product, often referred to as the Yat-kernel (Bouhsine, 2025). The resulting interactions are geometry-aware and inspired by inverse-square interactions in physics. Our method, Spherical Linearized Attention with Yat Kernels (SLAY), constrains queries and keys to the unit sphere so that attention depends only on angular alignment. Using Bernstein's theorem, we express the spherical Yat-kernel as a nonnegative mixture of polynomial-exponential product kernels and derive a strictly positive random-feature approximation enabling linear-time O(L) attention. We establish positive definiteness and boundedness on the sphere and show that the estimator yields well-defined, nonnegative attention scores. Empirically, SLAY achieves performance that is nearly indistinguishable from standard softmax attention while retaining linear time and memory scaling, and consistently outperforms prior linear-time attention mechanisms such as Performers and Cosformers. To the best of our knowledge, SLAY represents the closest linear-time approximation to softmax attention reported to date, enabling scalable Transformers without the typical performance trade-offs of attention linearization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。