提出线性化2-单纯形注意力，实现长序列全局建模且计算量线性增长。
Linearized 2-Simplicial Attention
- 将三重得分重写为复合查询与键的内积，使求和形式类比普通softmax注意力。
- 用正随机特征近似求和，固定大小状态存储历史，窗口内保留近期显式计算。
- 在16k上下文下优于混合模型，算力相当下下游平均准确率最高。
我们通过将三重得分重写为复合查询与键的内积，提出了2-单纯形注意力的线性化形式，使得对一个标记轴的求和具有与普通softmax注意力相同的结构。随后，我们利用正随机特征近似该求和，并以固定大小的状态存储全部历史，而第二个轴仍显式地处理一个短期窗口内的最近标记。这使得模型在序列长度上实现线性复杂度，同时具备窗口化2-单纯形注意力所缺乏的全局感知能力。我们使用自定义Triton内核实现该方法，并与Kimi Delta Attention结合，构建了一个完全不使用softmax注意力的模型。在匹配算力条件下，该模型在对比架构中实现了最高的平均下游准确率；在16k上下文长度下，其平均准确率超越了KDA混合模型，同时将LAMBADA困惑度从715.6降至602.6。
原文摘要 · Abstract (English)
We present a linearized form of 2-simplicial attention by rewriting the trilinear score as an inner product between a composite query and a key, so that the sum over one token axis takes the same form as ordinary softmax attention. We then approximate this sum with positive random features and store the entire past in a fixed-size state, while the second axis stays explicit over a short window of recent tokens. This enables us to achieve linear cost in sequence length combined with a global reach that windowed 2-simplicial attention lacks. We implement it with custom Triton kernels and combine it with Kimi Delta Attention to build a model with no softmax attention at all. Under matched compute, this model achieves the highest mean downstream accuracy among the compared architectures, and at 16k context it improves mean accuracy over a KDA hybrid while lowering LAMBADA perplexity from 715.6 to 602.6.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。