arXiv:2512.02403cs.LGcs.AR2025-12被引 1

提出利用局部相似性实现端到端稀疏加速,显著降低Transformer计算开销。

ESACT: An End-to-End Sparse Accelerator for Compute-Intensive Transformers via Local Similarity

  • 通过局部相似性预测机制提前估算注意力稀疏性,实现全组件稀疏化
  • 在26个基准上减少52.03%计算量,精度损失低于1%
  • 硬件层面优化,能效达3.29 TOPS/W,优于现有最先进加速器

Transformers由QKV生成、注意力计算和前馈网络(FFNs)构成,因其卓越性能已成为多领域主流模型。然而其高计算成本限制了高效硬件部署。稀疏性提供了可行方案,但现有加速器多仅利用注意力中的行内稀疏性,较少考虑行间稀疏性。依赖全局相似性估计的方法计算开销大,且通常只应用于一两个组件。通过分析注意力分布与计算流程,我们发现局部相似性可支持端到端稀疏加速且计算开销低。基于此提出ESACT,一种面向计算密集型Transformer的端到端稀疏加速器。核心为基于HLog量化实现局部相似性的稀疏性预测(SPLS)机制,可在QK生成前准确预测局部注意力稀疏性,实现所有组件高效稀疏化。为支持硬件实现,引入三项架构创新。26个基准实验表明,SPLS将总计算量减少52.03%,精度损失小于1%。ESACT达到3.29 TOPS/W的端到端能效,相比SOTA注意力加速器SpAtten和Sanger分别提升2.95倍和2.26倍的注意力层级能效。

原文摘要 · Abstract (English)

Transformers, composed of QKV generation, attention computation, and FFNs, have become the dominant model across various domains due to their outstanding performance. However, their high computational cost hinders efficient hardware deployment. Sparsity offers a promising solution, yet most existing accelerators exploit only intra-row sparsity in attention, while few consider inter-row sparsity. Approaches leveraging inter-row sparsity often rely on costly global similarity estimation, which diminishes the acceleration benefits of sparsity, and typically apply sparsity to only one or two transformer components. Through careful analysis of the attention distribution and computation flow, we observe that local similarity allows end-to-end sparse acceleration with lower computational overhead. Motivated by this observation, we propose ESACT, an end-to-end sparse accelerator for compute-intensive Transformers. ESACT centers on the Sparsity Prediction with Local Similarity (SPLS) mechanism, which leverages HLog quantization to accurately predict local attention sparsity prior to QK generation, achieving efficient sparsity across all transformer components. To support efficient hardware realization, we introduce three architectural innovations. Experimental results on 26 benchmarks demonstrate that SPLS reduces total computation by 52.03% with less than 1% accuracy loss. ESACT achieves an end-to-end energy efficiency of 3.29 TOPS/W, and improves attention-level energy efficiency by 2.95x and 2.26x over SOTA attention accelerators SpAtten and Sanger, respectively.

Transformer稀疏加速能效优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。