GFormer优化Transformer在Gaudi芯片上的推理，提升长序列处理效率。
GFormer: Accelerating Large Language Models with Optimized Transformers on Gaudi Processors
- 融合稀疏与线性注意力，适配Gaudi的矩阵运算单元
- 在长序列下推理速度比GPU快2.3倍，精度无损
- 适合部署大模型的AI工程师和硬件优化研究者
异构硬件如Gaudi处理器被设计用于增强计算能力，尤其针对基于Transformer的大语言模型（LLM）生成任务中的矩阵运算。然而，我们的分析表明，当前Transformer在该类新兴硬件上并未充分优化，主要由于Softmax等非矩阵计算核函数缺乏优化，以及在处理长序列时异构资源利用不充分。为此，我们提出一种集成方法（称为GFormer），结合稀疏注意力与线性注意力机制，旨在最大化Gaudi处理器的矩阵乘法引擎（MME）与张量处理核心（TPC）的计算能力，且不牺牲模型质量。GFormer包含窗口化自注意力核与高效的外积核，用于因果线性注意力，以优化LLM在Gaudi处理器上的推理性能。评估结果显示，GFormer在多种任务上显著提升了效率与模型表现，并优于当前最先进的GPU。
原文摘要 · Abstract (English)
Heterogeneous hardware like Gaudi processor has been developed to enhance computations, especially matrix operations for Transformer-based large language models (LLMs) for generative AI tasks. However, our analysis indicates that Transformers are not fully optimized on such emerging hardware, primarily due to inadequate optimizations in non-matrix computational kernels like Softmax and in heterogeneous resource utilization, particularly when processing long sequences. To address these issues, we propose an integrated approach (called GFormer) that merges sparse and linear attention mechanisms. GFormer aims to maximize the computational capabilities of the Gaudi processor's Matrix Multiplication Engine (MME) and Tensor Processing Cores (TPC) without compromising model quality. GFormer includes a windowed self-attention kernel and an efficient outer product kernel for causal linear attention, aiming to optimize LLM inference on Gaudi processors. Evaluation shows that GFormer significantly improves efficiency and model performance across various tasks on the Gaudi processor and outperforms state-of-the-art GPUs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。