通过递进选择减少Transformer注意力计算量，提升推理效率。
Cascade Token Selection for Transformer Attention Acceleration
- 跨层复用代表性令牌集，仅需少量增删更新。
- 在多个模型上实现22%~63%的矩阵运算节省，重叠率超0.8。
- 适合需要高效推理的大型语言模型部署场景。
本文提出一种级联令牌选择机制，用于降低Transformer注意力层中代表性令牌选择的计算开销。传统方法如激活去相关注意力（ADA）在每层需计算 $T imes T$ 的格拉姆矩阵以选择 $r ackslashackslash T$ 个代表令牌，而新方法将前一层的代表集继承至下一层，通过 $(T - r) imes r$ 的交叉格拉姆验证并仅做少量增删更新。该策略将每层选择成本从 $O(T^2 d)$ 降至 $O(T r d)$。在GPT-2 124M、GPT-J 6B和OPT 6.7B三个模型家族上，于AMD MI300X平台验证显示，格拉姆操作节省率达22%至63%，连续层间平均雅各布重叠达0.83至0.94。结果表明，有意义的令牌集合是输入的结构性属性，能在网络深度中保持一致传播：同一组令牌在不同层均携带非冗余信息。
原文摘要 · Abstract (English)
A method is presented for reducing the cost of representative token selection in transformer attention layers by exploiting the coherence of the representative set across depth. Activation Decorrelation Attention (ADA) selects $r \ll T$ representative tokens at each layer via a Gram threshold and computes attention on the compressed $r \times r$ problem, but the selection requires a $T \times T$ Gram matrix at every layer. The cascade mechanism introduced here inherits the representative set from layer $l$ to layer $l+1$, validates it via a $(T - r) \times r$ cross-Gram computation, and updates it with a small number of additions and removals. The cost of the selection step drops from $O(T^2 d)$ to $O(T r d)$ per layer. Validation on three model families (GPT-2 124M, GPT-J 6B, OPT 6.7B) on AMD MI300X demonstrates Gram operation savings of $22\%$ to $63\%$ with mean Jaccard overlap of $0.83$ to $0.94$ between consecutive layers. The cascade reveals that the set of informative tokens is a structural property of the input that propagates coherently through the depth of the network: the same tokens carry the non-redundant information at layer $l$ and at layer $l+1$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。