让手机端扩散语言模型生成快17至42倍,同时保持质量。
Efficient On-Device Diffusion LLM Inference with Mobile NPU

- 通过推测性解码填补后期计算空洞,提升NPU利用率。
- 双路径渐进修订机制避免令牌停滞,减少延迟。
- 优化内存布局与数据重映射,降低传输开销,适合移动端部署。
扩散大语言模型(dLLMs)通过并行去噪多个标记加速生成,适用于对延迟敏感的移动推理场景。然而,重复去噪在智能手机上引入了大量计算开销。移动神经处理单元(NPUs)具备高吞吐的密集矩阵计算能力,但高效利用仍具挑战:令牌确定后每块有效工作量缩小,令牌修订复杂化键值缓存复用,且有限的NPU可见地址空间导致昂贵的重映射和数据传输开销。本文提出llada.cpp,首个面向NPU的dLLM推理框架。通过三项技术实现块级推理与移动NPU执行特性的对齐:(1) 多块推测解码在当前块后期解码中填入推测的未来块标记,填补工作量萎缩;(2) 双路径渐进修订使已提交标记可修正直至稳定,并通过CPU路径刷新不稳定的标记,避免阻塞密集NPU执行;(3) 交换优化内存运行时压缩NPU可见地址布局,将数据预载与NPU计算重叠,减少重映射与传输开销。我们在多种硬件平台和dLLM负载上实现了llada.cpp的端到端框架评估。在使用前缀键值缓存复用的情况下,llada.cpp将LLaDA-8B生成延迟降低17至42倍,同时保持生成质量。
原文摘要 · Abstract (English)
Diffusion large language models (dLLMs) accelerate generation by denoising multiple tokens in parallel, making them attractive for latency-sensitive mobile inference. However, repeated denoising introduces substantial computation on smartphones. Mobile neural processing units (NPUs) offer high-throughput dense matrix computation, but efficiently exploiting them remains challenging: token commitment shrinks per-block effective workloads, token revision complicates KV cache reuse, and limited NPU-visible address space incurs costly remapping and data transfer overheads. In this paper, we propose llada.cpp, the first NPU-aware inference framework for accelerating dLLMs on smartphones. llada.cpp aligns block-wise dLLM inference with the execution characteristics of mobile NPUs through three techniques. (1) Multi-Block Speculative Decoding fills the shrinking workload in late-stage current-block decoding with speculative future-block tokens. (2) Dual-Path Progressive Revision keeps committed tokens revisable until stable and refreshes unstable tokens through a CPU-side path without stalling dense NPU execution. (3) Swap-Optimized Memory Runtime compacts NPU-visible address layouts and overlaps data staging with NPU computation to reduce remapping and transfer overheads. We implement llada.cpp as an end-to-end framework and evaluate it across diverse hardware platforms and dLLM workloads. llada.cpp reduces LLaDA-8B generation latency by 17x-42x over the CPU baseline with prefix KV cache reuse, while preserving generation quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。