软硬协同设计实现整数仅用的低精度Softmax，大幅降低能耗延迟。
SoftmAP: Software-Hardware Co-design for Integer-Only Softmax on Associative Processors
- 软硬协同设计，纯整数实现低精度Softmax
- 能效延迟积提升达三个数量级，优于A100与RTX3090
- 适合资源受限设备部署大型语言模型
近期研究致力于降低大语言模型（LLMs）的计算与内存开销，以使其能在资源受限设备上运行。尽管压缩技术取得进展，但非线性算子如Softmax和Layernorm仍因对量化敏感而成为瓶颈。我们提出SoftmAP，一种软硬协同设计方法，基于内存内计算（IMC）硬件实现纯整数低精度Softmax。该方法在能效延迟积上相较A100和RTX3090 GPU最高提升三个数量级，使大语言模型在不损失性能的前提下更易部署。
原文摘要 · Abstract (English)
Recent research efforts focus on reducing the computational and memory overheads of Large Language Models (LLMs) to make them feasible on resource-constrained devices. Despite advancements in compression techniques, non-linear operators like Softmax and Layernorm remain bottlenecks due to their sensitivity to quantization. We propose SoftmAP, a software-hardware co-design methodology that implements an integer-only low-precision Softmax using In-Memory Compute (IMC) hardware. Our method achieves up to three orders of magnitude improvement in the energy-delay product compared to A100 and RTX3090 GPUs, making LLMs more deployable without compromising performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。