首个面向移动端大模型的实时延迟分析工具，助力优化本地运行性能。
lm-Meter: Unveiling Runtime Inference Latency for On-Device Language Models
- 设计轻量级在线探针，实时追踪大模型推理各阶段延迟。
- 在最严苛电源模式下，预填充与解码阶段吞吐下降不足3%和1%。
- 揭示移动端推理瓶颈，适合模型优化与边缘部署开发者使用。
大型语言模型（LLMs）正日益融入日常应用，但其主流云端部署引发数据隐私与长期可持续性担忧。在手机和边缘设备上本地运行（on-device LLMs）可提升隐私性、可靠性并降低通信成本。然而，由于内存与算力需求大，且资源受限硬件上的性能-效率权衡不透明，实现该愿景仍具挑战。本文提出 lm-Meter，首个专为移动端 LLM 推理设计的轻量级、在线延迟分析工具。它无需外部设备，即可在阶段（如嵌入、prefill、decode、softmax、sampling）与内核级别捕获细粒度实时延迟。我们在商用移动平台实现 lm-Meter，证明其高精度与极低系统开销：在最严苛的 Powersave 调度器下，prefill 阶段吞吐仅下降 2.58%，decode 阶段下降 0.99%。基于 lm-Meter，我们开展全面实证研究，揭示了移动端推理中的阶段与内核级瓶颈，量化了准确率-效率权衡，并发现系统性优化机会。lm-Meter 为受限平台上的 LLM 运行行为提供了前所未有的可观测性，为智能优化奠定基础，推动 on-device LLM 系统普及。代码与教程见 https://github.com/amai-gsu/LM-Meter。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly integrated into everyday applications, but their prevalent cloud-based deployment raises growing concerns around data privacy and long-term sustainability. Running LLMs locally on mobile and edge devices (on-device LLMs) offers the promise of enhanced privacy, reliability, and reduced communication costs. However, realizing this vision remains challenging due to substantial memory and compute demands, as well as limited visibility into performance-efficiency trade-offs on resource-constrained hardware. We propose lm-Meter, the first lightweight, online latency profiler tailored for on-device LLM inference. lm-Meter captures fine-grained, real-time latency at both phase (e.g., embedding, prefill, decode, softmax, sampling) and kernel levels without auxiliary devices. We implement lm-Meter on commercial mobile platforms and demonstrate its high profiling accuracy with minimal system overhead, e.g., only 2.58% throughput reduction in prefill and 0.99% in decode under the most constrained Powersave governor. Leveraging lm-Meter, we conduct comprehensive empirical studies revealing phase- and kernel-level bottlenecks in on-device LLM inference, quantifying accuracy-efficiency trade-offs, and identifying systematic optimization opportunities. lm-Meter provides unprecedented visibility into the runtime behavior of LLMs on constrained platforms, laying the foundation for informed optimization and accelerating the democratization of on-device LLM systems. Code and tutorials are available at https://github.com/amai-gsu/LM-Meter.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。