通过将部分计算移至CPU，显著提升大模型在线推理的吞吐量。
NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference
- 将注意力计算和键值缓存卸载到本地CPU，缓解GPU内存瓶颈。
- 在T4/A10G/H100上分别实现最高7.5倍、26%、14%的吞吐提升。
- 适合资源受限但拥有较强CPU的部署场景，如边缘或低成本服务器。
在线大模型推理支撑了智能聊天机器人和自主代理等众多应用。当前主流推理引擎依赖请求批处理以提高吞吐量，使昂贵的GPU加速更高效。然而，有限的GPU内存严重限制了实际可达到的批大小，导致大量GPU计算资源闲置。本文提出NEO，一种在线大模型推理系统，将部分注意力计算和键值缓存状态从GPU卸载至本地主机CPU，有效提升GPU批大小与推理吞吐量。为此，NEO设计了不对称的GPU-CPU流水线与负载感知调度策略，平衡两者负载，充分挖掘计算与内存资源。我们在多种工作负载（代码生成、文本摘要）、GPU（T4、A10G、H100）和模型（7B、8B、70B）上评估了NEO。相比纯GPU方案，NEO在T4、A10G、H100上分别实现最高7.5×、26%、14%的吞吐提升，同时保持相同延迟；当使用更强的CPU时，A10G上可达79.3%的吞吐增益。
原文摘要 · Abstract (English)
Online LLM inference powers many exciting applications such as intelligent chatbots and autonomous agents. Modern LLM inference engines widely rely on request batching to improve inference throughput, aiming to make it cost-efficient when running on expensive GPU accelerators. However, the limited GPU memory has largely limited the batch size achieved in practice, leaving significant GPU compute resources wasted. We present NEO, an online LLM inference system that offloads part of attention compute and KV cache states from the GPU to the local host CPU, effectively increasing the GPU batch size and thus inference throughput. To this end, NEO proposes asymmetric GPU-CPU pipelining and load-aware scheduling to balance GPU and CPU loads and fully utilize their compute and memory resources. We evaluate NEO on a wide range of workloads (i.e., code generation, text summarization), GPUs (i.e., T4, A10G, H100), and LLM models (i.e., 7B, 8B, 70B). NEO achieves up to 7.5$\times$, 26%, and 14% higher throughput compared to GPU-only approach on T4, A10G, and H100 GPUs, respectively, while maintaining the same latency; with more powerful CPUs, NEO achieves up to 79.3% throughput gain on A10G GPU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。