用外接协处理器提升大批次推理效率，降低能耗。
HPU: High-Bandwidth Processing Unit for Scalable, Cost-effective LLM Inference via GPU Co-processing
- 用协处理器卸载内存密集型计算，让GPU专注算力任务。
- 实测性能提升4.1倍，能效提高4.6倍，不增GPU数量。
- 适合需要高吞吐、低成本部署的大模型推理场景。
基于Transformer的大模型注意力层因运算强度低且需大量存储键值缓存，导致当前GPU系统效率低下。本文提出一种高带宽协处理器（HPU），作为内存密集型计算的外挂单元，通过卸载内存密集型操作，使GPU专注于计算密集型任务，从而提升整体资源利用率。该HPU以基于PCIe的FPGA卡形式集成于GPU系统，可随大批次和长序列需求弹性扩展。实验表明，所提出的GPU-HPU异构系统相比纯GPU方案，最高实现4.1倍性能提升与4.6倍能效改进，在不增加GPU数量的前提下实现可扩展性。
原文摘要 · Abstract (English)
The attention layer, a core component of Transformer-based LLMs, brings out inefficiencies in current GPU systems due to its low operational intensity and the substantial memory requirements of KV caches. We propose a High-bandwidth Processing Unit (HPU), a memoryintensive co-processor that enhances GPU resource utilization during large-batched LLM inference. By offloading memory-bound operations, the HPU allows the GPU to focus on compute-intensive tasks, increasing overall efficiency. Also, the HPU, as an add-on card, scales out to accommodate surging memory demands driven by large batch sizes and extended sequence lengths. In this paper, we show the HPU prototype implemented with PCIe-based FPGA cards mounted on a GPU system. Our novel GPU-HPU heterogeneous system demonstrates up to 4.1x performance gains and 4.6x energy efficiency improvements over a GPUonly system, providing scalability without increasing the number of GPUs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。