利用边缘设备加速大模型推理，显著降低延迟和成本。
SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMs
- 将大模型推理拆分至边缘与服务器，仅传输生成的词元。
- 服务器吞吐量提升2.22倍，整体成本效率提高1.91倍。
- 适合需要低延迟、高并发的大模型交互应用。
大型语言模型（LLMs）驱动众多现代应用，但其大规模部署仍成本高昂且资源密集。现有以服务器为中心的系统忽视了边缘端的消费级GPU。我们提出SpecEdge，一种基于推测解码的边缘辅助推理框架，将LLM工作负载在边缘与服务器GPU间分配，仅通过网络交换词元输出。SpecEdge采用主动边缘预写机制，使边缘生成词元与服务器验证重叠，并使用感知流水线的调度策略，交错处理多个用户请求以提升服务器吞吐量。实验表明，SpecEdge相比纯服务器基线，整体成本效率提升1.91倍，实现2.22倍的服务器吞吐量提升，同时将词元间延迟降低11.24%，为大模型服务提供了一种可扩展、低成本的新范式。代码已开源于https://github.com/kaist-ina/specedge。
原文摘要 · Abstract (English)
Large language models (LLMs) power many modern applications, but serving them at scale remains costly and resource-intensive. Current server-centric systems overlook consumer-grade GPUs at the edge. We introduce SpecEdge, an edge-assisted inference framework that splits LLM workloads between edge and server GPUs using a speculative decoding scheme, exchanging only token outputs over the network. SpecEdge employs proactive edge drafting to overlap edge token creation with server verification and pipeline-aware scheduling that interleaves multiple user requests to increase server-side throughput. Experiments show SpecEdge enhances overall cost efficiency by 1.91x through achieving 2.22x server throughput, and reduces inter token latency by 11.24% compared to a server-only baseline, introducing a scalable, cost-effective paradigm for LLM serving. The code is available at https://github.com/kaist-ina/specedge
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。