通过低秩分解加速Transformer推理，不改结构也不降精度。
Gated Subspace Inference for Transformer Acceleration
- 将激活向量拆分为低秩子空间与残差，仅对子空间计算线性层。
- 在三类模型上实现3.0x~10.5x加速，困惑度几乎不变，准确率超98%。
- 无需重训练或修改架构，适合部署在资源受限设备上。
本文提出一种加速Transformer语言模型推理的方法，利用每层令牌激活流形的低有效秩特性。该方法将每个激活向量分解为子空间分量和残差，通过缓存的低秩权重图像在降低内存带宽的前提下计算线性层输出，并引入逐令牌门控机制，决定是否计算残差修正。门控确保输出分布保持在可控误差范围内。在AMD MI300X上对GPT-2 124M、GPT-J 6B和OPT 6.7B三个模型族的验证表明，线性层权重读取速度提升3.0x至10.5x，困惑度比低于1.00，顶1词准确率高于98%。该方法无需重训练、无架构修改，也未近似注意力机制。在GPT-J 6B（k=256, ε=0.05）操作点下，加速模型与基线产生字符级完全一致的输出。
原文摘要 · Abstract (English)
A method is presented for accelerating inference in transformer language models by exploiting the low effective rank of the token activation manifold at each layer. The method decomposes each activation vector into a subspace component and a residual, computes the linear-layer output on the subspace component via a cached low-rank weight image at reduced memory bandwidth, and applies a per-token gate that determines whether the residual correction is computed or skipped. The gate ensures that the output distribution is preserved to within a controllable tolerance. Validation on three model families (GPT-2 124M, GPT-J 6B, OPT 6.7B) on AMD MI300X demonstrates effective speedups of 3.0x to 10.5x on linear-layer weight reads with perplexity ratios below 1.00 and top-1 token agreement above 98%. The method requires no retraining, no architectural modification, and no approximation of the attention mechanism. At the operating point (k = 256, ε = 0.05) on GPT-J 14 6B, the accelerated model produces character-for-character identical output to the baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。