将大模型推理的两阶段拆分到同一显卡，提升资源利用率。
Splitwiser: Efficient LM inference with constrained resources
- 把提示词计算和生成阶段分在同个GPU上运行
- 减少数据传输开销，提升内存访问效率
- 适用于资源受限场景下的高效推理
大模型高效推理仍是关键挑战，主要分为计算密集的提示词处理和内存密集的令牌生成两个阶段。尽管已有批处理与调度技术，但令牌生成阶段仍未能充分使用计算资源，尤其相较于提示词阶段。为此，我们提出Splitwiser方法，将一个大模型推理请求的两个阶段拆分至同一GPU上执行，从而降低开销并提升内存访问与缓存利用率。通过消除跨设备数据传输，旨在最小化网络相关开销。本文描述了所提流水线的基本结构，并分享初步结果与分析。我们在Huggingface和vLLM两种广泛使用的独立LLM架构上实现了该多进程设计，并开源了相应代码：1）Huggingface版本（https://github.com/asad-aali/splitwiser），2）vLLM版本（https://github.com/adney11/vllm-sysml）。
原文摘要 · Abstract (English)
Efficient inference of LLMs remains a crucial challenge, with two main phases: a compute-intensive prompt computation and a memory-intensive token generation. Despite existing batching and scheduling techniques, token generation phases fail to fully utilize compute resources, especially when compared to prompt computation phases. To address these challenges, we propose Splitwiser, a methodology that splits the two phases of an LLM inference request onto the same GPU, thereby reducing overhead and improving memory access and cache utilization. By eliminating the need to transfer data across devices, Splitwiser aims to minimize network-related overheads. In this report, we describe the basic structure of our proposed pipeline while sharing preliminary results and analysis. We implement our proposed multiprocessing design on two widely-used and independent LLM architectures: Huggingface and vLLM. We open-source our code for the respective implementations: 1) Huggingface (https://github.com/asad-aali/splitwiser), and 2) vLLM (https://github.com/adney11/vllm-sysml).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。