用新型存内计算芯片加速大模型注意力，速度提升百倍、功耗降千倍。
Analog In-Memory Computing Attention Mechanism for Fast and Energy-Efficient Large Language Models
- 用电荷型存内计算单元实现并行模拟点积运算，避免数据搬运
- 相比GPU，注意力延迟降低两个数量级，能耗降低五个数量级
- 无需重新训练即可达到GPT-2性能，适合低功耗部署场景
Transformer网络依赖自注意力机制驱动大语言模型。在生成式Transformer中，自注意力通过缓存记忆存储标记投影，避免每步重复计算。然而，GPU存储的投影需每次加载至SRAM，导致延迟和能效瓶颈。本文提出基于新兴电荷型存储器（增益单元）的自注意力存内计算架构，可高效写入新标记并实现自注意力所需的并行模拟点积计算。为应对模拟电路非理想性带来的映射难题，我们设计了一种初始化算法，在无需从头训练的情况下实现了与GPT-2相当的文本处理性能。该架构相较GPU，分别将注意力延迟和能耗降低最多两个和五个数量级，标志着向超快、低功耗生成式Transformer迈出了关键一步。
原文摘要 · Abstract (English)
Transformer networks, driven by self-attention, are central to Large Language Models. In generative Transformers, self-attention uses cache memory to store token projections, avoiding recomputation at each time step. However, GPU-stored projections must be loaded into SRAM for each new generation step, causing latency and energy bottlenecks. We present a custom self-attention in-memory computing architecture based on emerging charge-based memories called gain cells, which can be efficiently written to store new tokens during sequence generation and enable parallel analog dot-product computation required for self-attention. However, the analog gain cell circuits introduce non-idealities and constraints preventing the direct mapping of pre-trained models. To circumvent this problem, we design an initialization algorithm achieving text processing performance comparable to GPT-2 without training from scratch. Our architecture respectively reduces attention latency and energy consumption by up to two and five orders of magnitude compared to GPUs, marking a significant step toward ultra-fast, low-power generative Transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。