LeanStream通过推测与精炼实现高效移动端大模型推理，显著降低内存占用。
LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference
- 采用推测-精炼流水线，利用部分GPU结果动态调整计算与加载优先级
- 内存占用减少4.8至7.5倍，生成吞吐提升1.6至2.1倍
- 适合资源受限设备上的大模型实时推理，尤其关注隐私与响应速度
在移动和嵌入式设备上进行大模型（LLM）推理对保护隐私和保证响应速度具有吸引力，但因模型权重远超可用的DRAM容量而面临挑战。现有系统利用激活稀疏性，并将权重卸载至SSD或闪存存储，但存在根本性的系统权衡：准确的稀疏执行决策需要最新上下文，而高效的计算与I/O重叠则要求提前预测。因此，现有设计要么串行执行，要么导致冗余权重读取、额外计算和大的缓存开销。我们提出LeanStream，一种用于高效设备端LLM推理的流式推测-精炼框架。LeanStream通过部分GPU结果逐步优化计算、加载和缓存保留优先级，实现细粒度的GPU执行与存储I/O重叠。我们在移动和嵌入式平台均实现了LeanStream。相较于之前的设备端LLM推理系统，LeanStream在先前工作最佳吞吐下的内存使用量减少了4.8至7.5倍，同时进一步将令牌生成吞吐提升1.6至2.1倍。
原文摘要 · Abstract (English)
On-device LLM inference is attractive for privacy and responsiveness, but remains challenging on mobile and embedded devices because model weights far exceed available DRAM. Prior systems exploit activation sparsity and offload weights to SSD or flash storage, but face a fundamental systems trade-off: accurate sparse execution decisions require the latest context, whereas efficient computation-I/O overlap requires early prediction. As a result, existing designs either serialize execution or incur redundant weight fetches, extra computation, and large cache overheads. We present LeanStream, a streaming speculate-and-refine framework for efficient on-device LLM inference. LeanStream progressively refines computation, loading, and cache-retention priorities using partial GPU results, enabling fine-grained overlap between GPU execution and storage I/O. We implement LeanStream on both mobile and embedded platforms. Compared with prior on-device LLM inference systems, LeanStream reduces memory usage by 4.8$\times$ to 7.5$\times$ at the best throughput achieved by prior work, while further improving token generation throughput by 1.6$\times$ to 2.1$\times$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。