多设备协同缓存提示，显著加速边缘端小模型推理
Accelerating Local LLMs on Resource-Constrained Edge Devices via Distributed Prompt Caching
- 多台低配设备共享中间计算状态，利用提示相似性提升效率
- 在树莓派零2W上实现首令牌耗时降低93.12%，末令牌耗时降低50.07%
- 用布隆过滤器减少无效通信，适合资源受限的边缘部署场景
由于本地大模型在资源受限的边缘设备上推理存在严重性能瓶颈，本文提出分布式提示缓存机制，通过多台低功耗边缘设备协作共享中间处理状态，以提升推理效率。为充分挖掘提示间的相似性，该机制支持部分匹配。由于跨无线网络共享状态会引入通信开销，我们引入基于布隆过滤器的数据结构——目录（catalog），用于判断远程服务器是否拥有所需内部状态，从而抑制不必要的通信。在树莓派零2W平台上，使用Gemma-3 270M模型和MMLU数据集的实验表明，所提方法平均将首令牌时间（TTFT）降低93.12%，末令牌时间（TTLT）降低50.07%。
原文摘要 · Abstract (English)
Since local LLM inference on resource-constrained edge devices imposes a severe performance bottleneck, this paper proposes distributed prompt caching to enhance inference performance by cooperatively sharing intermediate processing states across multiple low-end edge devices. To fully utilize prompt similarity, our distributed caching mechanism also supports partial matching. As this approach introduces communication overhead associated with state sharing over a wireless network, we introduce a Bloom-filter-based data structure, referred to as a catalog, to determine whether a remote server possesses the desired internal states, thereby suppressing unnecessary communication. Experiments using the Gemma-3 270M model and the MMLU dataset on the Raspberry Pi Zero 2W platform demonstrate that the proposed approach reduces TTFT (Time to First Token) and TTLT (Time to Last Token) by 93.12% and 50.07% on average, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。