通过智能调度减少设备端大模型推理延迟和能耗
SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference
- 按代价动态决定缓存块是云端流式加载还是本地计算
- 首字延迟降低1.3倍至5.1倍，能耗下降1.5至3.3倍
- 适合资源受限的移动设备部署，适应网络波动
由于硬件资源有限以及预填充阶段需处理完整输入上下文以构建键值（KV）缓存，导致设备端大语言模型高效推理仍具挑战。我们提出SparKV，一种结合云端KV流式传输与本地计算的自适应缓存加载框架。SparKV对每个KV块的开销建模，并决定其应通过流式传输或本地计算；同时重叠两种执行路径以降低延迟。为应对无线连接波动和边缘资源变化，SparKV在运行时动态优化离线生成的调度方案，平衡通信与计算开销。在多种数据集、大模型和边缘设备上的实验表明，SparKV将首字延迟降低1.3×至5.1×，响应质量几乎无损，且单请求能耗降低1.5×至3.3×，展现出良好的鲁棒性与实际部署价值。
原文摘要 · Abstract (English)
Efficient inference for on-device Large Language Models (LLMs) remains challenging due to limited hardware resources and the high cost of the prefill stage, which processes the full input context to construct Key-Value (KV) caches. We present SparKV, an adaptive KV loading framework that combines cloud-based KV streaming with on-device computation. SparKV models the cost of individual KV chunks and decides whether each chunk should be streamed or computed locally, while overlapping the two execution paths to reduce latency. To handle fluctuations in wireless connectivity and edge resource availability, SparKV further refines offline-generated schedules at runtime to rebalance communication and computation costs. Experiments across diverse datasets, LLMs, and edge devices show that SparKV reduces Time-to-First-Token by 1.3$x-5.1x with negligible impact on response quality, while lowering per-request energy consumption by 1.5x to 3.3x, demonstrating its robustness and practicality for real-world on-device deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。