通过流水线卸载提升消费级设备上大模型推理效率
PIPO: Pipelined Offloading for Efficient Inference on Consumer Devices
- 设计细粒度流水线卸载机制，优化数据传输与计算
- 在6GB显存设备上将GPU利用率从不足40%提升至90%以上
- 适合资源受限的终端设备部署大模型推理任务
大型语言模型（LLMs）对内存和计算需求高，难以在消费级设备上部署，因其受限于GPU内存。模型卸载可缓解内存压力，但常导致GPU利用率低，影响推理效率。本文提出一种名为流水线卸载（PIPO）的新框架，通过细粒度卸载流水线、优化的数据传输与计算调度，实现高并发与高效推理。实验结果表明，相较于当前最优基线，PIPO将GPU利用率从低于40%提升至超过90%，吞吐量最高提升3.1倍，测试环境为配备6GB显存RTX3060显卡的笔记本电脑。
原文摘要 · Abstract (English)
The high memory and computation demand of large language models (LLMs) makes them challenging to be deployed on consumer devices due to limited GPU memory. Offloading can mitigate the memory constraint but often suffers from low GPU utilization, leading to low inference efficiency. In this work, we propose a novel framework, called pipelined offloading (PIPO), for efficient inference on consumer devices. PIPO designs a fine-grained offloading pipeline, complemented with optimized data transfer and computation, to achieve high concurrency and efficient scheduling for inference. Experimental results show that compared with state-of-the-art baseline, PIPO increases GPU utilization from below 40% to over 90% and achieves up to 3.1$\times$ higher throughput, running on a laptop equipped with a RTX3060 GPU of 6GB memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。