利用推测解码释放被闲置的GPU算力，提升低资源设备上大模型推理效率。
SpecOffload: Unlocking Latent GPU Capacity for LLM Inference on Resource-Constrained Devices
- 将推测解码嵌入卸载流程，用闲置GPU内存存执行草稿模型。
- 相比最佳基线，GPU核心利用率提升4.49倍，吞吐量提高2.54倍。
- 适合在内存受限设备上部署大模型推理，尤其对实时性要求高的场景。
资源受限设备上的高效大模型推理面临计算与内存利用的双重挑战。由于GPU内存有限，现有系统将模型权重卸载至CPU内存，导致CPU与GPU间产生显著的I/O开销。这造成两大效率瓶颈：(1) GPU核心常处于空闲状态，等待数据加载；(2) GPU内存容量对性能影响甚微，减少其容量几乎不影响整体吞吐量。本文提出SpecOffload，一种高吞吐推理引擎，将推测解码融入卸载流程。核心思想是利用潜在闲置的GPU资源存储并执行用于推测解码的草稿模型，从而在近乎零额外成本下加速推理。为此，我们精心协调目标模型与草稿模型在推测解码中的交错执行，并设计调度器管理张量布局与参数选择。相较于最佳基线，SpecOffload将GPU核心利用率提升4.49倍，推理吞吐量提升2.54倍。代码已开源：https://github.com/MobiSense/SpecOffload-public。
原文摘要 · Abstract (English)
Efficient LLM inference on resource-constrained devices presents significant challenges in compute and memory utilization. Due to limited GPU memory, existing systems offload model weights to CPU memory, incurring substantial I/O overhead between the CPU and GPU. This leads to two major inefficiencies: (1) GPU cores are underutilized, often remaining idle while waiting for data to be loaded; and (2) GPU memory has low impact on performance, as reducing its capacity has minimal effect on overall throughput.In this paper, we propose SpecOffload, a high-throughput inference engine that embeds speculative decoding into offloading. Our key idea is to unlock latent GPU resources for storing and executing a draft model used for speculative decoding, thus accelerating inference at near-zero additional cost. To support this, we carefully orchestrate the interleaved execution of target and draft models in speculative decoding within the offloading pipeline, and propose a planner to manage tensor placement and select optimal parameters. Compared to the best baseline, SpecOffload improves GPU core utilization by 4.49x and boosts inference throughput by 2.54x. Our code is available at https://github.com/MobiSense/SpecOffload-public .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。