编译器辅助推测采样，让边缘设备更快生成大模型文本。
Compiler-Assisted Speculative Sampling for Accelerated LLM Inference on Heterogeneous Edge Devices
- 用编译器分析硬件成本，智能划分大模型计算任务。
- 在真实边缘设备上实现1.68倍加速，适合短输入场景。
- 特别适合资源受限的实时应用，如车载或可穿戴设备。
在资源受限的边缘设备上部署大语言模型面临严重延迟挑战，尤其在需要实时响应的应用中，延迟可能影响安全或可用性。针对传统逐词生成效率低的问题，推测解码（Speculative Decoding, SD）成为有前景的解决方案。然而，边缘端的SD面临两大难题：(1) 如何在不牺牲性能或可编程性的前提下，将SD集成到编译器工作流中；(2) 如何通过精心设计的划分策略，有效利用现代SoC的异构计算资源。本文提出一种分析性成本模型，用于探索异构硬件配置，并指导大模型子图的粗粒度划分，尤其适用于边缘常见的短输入序列。该模型能预测推测采样与异构执行联合带来的收益，在搭载六核Cortex-A CPU和Mali GPU的边缘设备上验证，翻译任务最高获得1.68×加速，结果与理论预期高度一致。
原文摘要 · Abstract (English)
LLM deployment on resource-constrained edge devices faces severe latency constraints, particularly in real-time applications where delayed responses can compromise safety or usability. Among many approaches to mitigate the inefficiencies of sequential token-by-token generation, Speculative Decoding (SD) has emerged as a promising technique. However, SD at the edge is hindered by two major challenges: (1) integrating SD into a compiler-based workflow without sacrificing performance or programmability, and (2) exploiting the heterogeneous compute resources of modern SoCs through carefully designed partitioning strategies. This work addresses these challenges by using an analytical cost model that explores heterogeneous hardware configurations and guides coarse-grained partitioning of LLM subgraphs, particularly with edge-typical short input sequence lengths. The cost model predicts when speculative sampling and heterogeneous execution are jointly beneficial and is validated on an edge device featuring a hexacore Cortex-A CPU and a Mali GPU, revealing up to 1.68$\times$ speedup for translation tasks, closely matching analytic expectations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。