OPAL通过保留异常值实现高效大模型生成加速。
OPAL: Outlier-Preserved Microscaling Quantization Accelerator for Generative Large Language Models
- 采用微缩格式量化激活值，每块保留4个异常值。
- 敏感层用5比特，非敏感层用3比特，混合精度提升效率。
- 硬件含浮点单元处理异常值，适合低功耗部署场景。
为应对大语言模型（LLMs）规模不断增长带来的内存和带宽压力，近期研究聚焦于权重的激进量化，但对激活量化的研究仍不足。本文提出一种软硬件协同设计方法，构建面向生成任务的高效能LLM加速器OPAL。首先，提出一种新型激活量化方法，采用微缩数据格式，并在每个子张量块中保留若干异常值（如128个元素中保留4个）。其次，在保留异常值基础上，引入混合精度策略：解码器块中敏感层输入使用5比特，其余层保持3比特。最后，设计了包含用于处理异常值的浮点单元和用于主要非异常操作的向量化整数乘法器的OPAL硬件架构。此外，采用基于log2的近似方法实现softmax运算，仅需移位与减法操作以最大化能效。实验结果表明，该方案可提升能效1.6~2.2倍，面积减少2.4~3.1倍，且精度损失极小，即困惑度增加低于1。
原文摘要 · Abstract (English)
To overcome the burden on the memory size and bandwidth due to ever-increasing size of large language models (LLMs), aggressive weight quantization has been recently studied, while lacking research on quantizing activations. In this paper, we present a hardware-software co-design method that results in an energy-efficient LLM accelerator, named OPAL, for generation tasks. First of all, a novel activation quantization method that leverages the microscaling data format while preserving several outliers per sub-tensor block (e.g., four out of 128 elements) is proposed. Second, on top of preserving outliers, mixed precision is utilized that sets 5-bit for inputs to sensitive layers in the decoder block of an LLM, while keeping inputs to less sensitive layers to 3-bit. Finally, we present the OPAL hardware architecture that consists of FP units for handling outliers and vectorized INT multipliers for dominant non-outlier related operations. In addition, OPAL uses log2-based approximation on softmax operations that only requires shift and subtraction to maximize power efficiency. As a result, we are able to improve the energy efficiency by 1.6~2.2x, and reduce the area by 2.4~3.1x with negligible accuracy loss, i.e., <1 perplexity increase.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。