arXiv:2605.16786cs.LG2026-05

让手机在闪存上高效运行大模型,推理速度提升近3倍。

Lever: Speculative LLM Inference on Smartphones

论文配图:Lever: Speculative LLM Inference on Smartphones
图 1 · 摘自论文原文
  • 用小模型在内存中预拟文本,大模型在闪存中验证,减少读写次数。
  • 通过剪枝低价值分支和优化硬件调度,平均降低2.93倍延迟。
  • 专为手机设计,适合资源受限的移动应用部署。

大型语言模型(LLMs)在交互式移动端应用中的需求日益增长,但高质量模型超出智能手机有限的DRAM容量。闪存可容纳更大模型,然而基于闪存的推理速度慢,因为自回归解码反复调用目标模型并引发昂贵的I/O操作。我们观察到,推测性解码天然适配此场景:小的草稿模型可保留在DRAM中,而更大的闪存驻留目标模型每次调用可验证多个候选词。然而,现有方法假设服务器级加速器,未考虑长期的I/O延迟、有限的计算并行性和不规则的推测执行。我们提出Lever,一个面向智能手机的高效闪存后端LLM推理端到端系统。Lever在移动端约束下联合优化推测解码的三个阶段:起草阶段,使用兼顾I/O与计算开销的增益-成本目标构建词元树;验证阶段,通过早期退出预测剪枝低价值分支以减少目标模型计算;执行阶段,将推测任务高效映射至移动端CPU-NPU硬件,提升利用率。全面评估显示,Lever相比基线闪存卸载推理平均降低2.93倍延迟,相比传统推测解码降低1.50倍延迟,显著缩小了闪存后端与内存驻留模型间的延迟差距。

原文摘要 · Abstract (English)

Large language models (LLMs) are increasingly needed for interactive mobile applications, but high-quality models exceed the limited DRAM available on smartphones. Flash storage can hold larger models, yet flash-backed inference is slow because autoregressive decoding repeatedly invokes the target model and incurs costly I/O. We observe that speculative decoding is a natural fit for this setting: a small draft model can remain in DRAM, while a larger flash-resident target model verifies multiple candidate tokens per invocation. However, existing methods assume server-class accelerators and fail to account for prolonged I/O latency, limited computation parallelism, and irregular speculation execution. We present Lever, an end-to-end system for efficient flash-backed LLM inference on smartphones. Lever jointly optimizes the three stages of speculative decoding under mobile constraints. For drafting, it builds token trees using an I/O- and compute-aware gain-cost objective. For verification, it prunes low-value branches through early-exit prediction to reduce target-model computation. For execution, it maps speculation efficiently across mobile CPU-NPU hardware to improve utilization. Comprehensive evaluations show that Lever reduces inference latency by an average of 2.93x over baseline flash-offloaded inference and 1.50x over conventional speculative decoding, narrowing the latency gap between flash-backed and memory-resident LLM inference.

大模型移动端推理优化闪存

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。