提出可线性化Transformer的分析方法，显著降低长文本推理开销。
The Key to Going Linear: Analysis-Driven Transformer Linearization
- 通过冻结主干网络分析状态更新设计，揭示软注意力机制的关键结构。
- 引入汇点令牌等结构干预，使320亿参数模型在MMLU上超越已有基线。
- 适合关注长序列推理加速与模型压缩的研究者与工程师。
因果自注意力的二次计算成本严重制约了长上下文Transformer的推理效率。尽管已有多种事后线性化方法，但难以判断哪些组件能保持模型性能。本文在严格冻结主干的前提下，隔离分析状态更新设计的影响。结果表明，软注意力依赖于基于键的秩-1正交投影，解释了为何增量式网络优于纯门控累积。我们识别出近似误差的潜在来源，并提出结构化干预策略，包括汇点令牌、短卷积和固定预算缓存路由，有效缩小剩余差距。该线性化方法在LLaMA与Qwen模型上扩展至32B参数规模，在MMLU上表现优于以往事后基线，并达到复杂自适应缓存框架的长上下文检索能力。
原文摘要 · Abstract (English)
The quadratic cost of causal self-attention severely bottlenecks long-context transformer inference. While numerous post hoc linearization pipelines exist, it is difficult to identify which components preserve model quality. This work isolates the effect of state update design in a strict frozen-backbone regime. We show that softmax relies on key-dependent, rank-1 orthogonal projections, elucidating why delta-style networks outperform purely gated accumulation. We identify a potential source of approximation errors and introduce structural interventions, specifically sink tokens, short convolutions, and fixed-budget cache routing, which reduces the remaining gap. We scale this linearization approach across LLaMA and Qwen models up to 32B parameters, outperforming prior post hoc baselines on MMLU and matching the long-context retrieval of complex adaptive-caching frameworks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。