新注意力机制通过排序实现近线性复杂度，保留强表达能力。
Sliced ReLU attention: Quasi-linear contextual expressivity via sorting
- 用键值差的投影排序替代软最大运算，结构更简单。
- 计算复杂度为O(n log n)，适合超长序列处理。
- 理论证明可完成复杂序列解耦任务，适合小中规模实验。
我们提出一种新型注意力机制——切片ReLU注意力，其结构不同于软最大及其近似方法。不将非线性作用于成对点积，而是对键-查询差的低维投影进行操作，并利用排序实现准线性复杂度。该构造生成可微、非对称的核函数，可通过排序在O(n log n)时间内计算，适用于极长上下文。除计算优势外，模型仍具备强大理论表达能力：我们建立了两项上下文表达性结果，此前仅知于软最大注意力，表明切片ReLU注意力保持了执行非平凡序列到序列解耦任务的能力，并满足上下文通用逼近性质。最后，我们在小至中等规模实验中展示了该核函数的实际潜力。
原文摘要 · Abstract (English)
We introduce sliced ReLU attention, a new attention mechanism that departs structurally from both softmax and its approximation alternatives. Instead of applying a nonlinearity to pairwise dot products, we operate on one-dimensional projections of key--query differences and leverage sorting to obtain quasi-linear complexity. This construction yields a differentiable, non-symmetric kernel that can be computed in O(n log(n)) through a sorting procedure, making it suitable for very long contexts. Beyond computational benefits, the model retains strong theoretical expressive power: we establish two in-context expressivity results, previously known for softmax attention, showing that sliced ReLU attention preserves the ability to perform nontrivial sequence-to-sequence disentangling tasks and satisfies a contextual universal approximation property. Finally, we illustrate the potential practical interest of this kernel in small to medium-scale experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。