arXiv:2608.13076cs.AI2026-08

用边缘端小模型预生成,云端大模型验证,大幅降低大模型推理成本。

SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference

论文配图:SPADE: Speculative Decoding for Precise and Low Cost Distributed Edge Cloud Inference
图 1 · 摘自论文原文
  • 边缘端用小模型快速生成候选词,云端大模型并行验证并筛选
  • 实验显示云调用减少76%且准确率零损失
  • 无需重训练,适合部署在资源受限的边缘场景

大型语言模型(LLMs)在自然语言理解与生成上取得显著进展,但其部署受高计算需求制约。直接在边缘部署小型模型可缓解算力压力,但会降低准确性;而依赖云端大模型虽保持性能,却带来高昂的按令牌计费成本。本文提出一种分布式推理框架 our{},在边缘与云端间集成推测性解码(SD)。边缘部署的紧凑型草稿模型快速生成候选令牌,云端的大验证模型并行验证这些令牌。被接受的令牌保留,仅拒绝时才触发验证器修正,显著减少云端调用次数。该即插即用设计将主要计算负载移至边缘,大幅降低推理时延与云成本,且无需任何重训练即可保持大模型精度。在SpecBench与CNN/Dailymail数据集上的多任务实验表明, our{} 相比完整模型减少76%的云模型调用,同时实现零准确率损失。

原文摘要 · Abstract (English)

Large Language Models (LLMs) have achieved remarkable success in natural language understanding and generation, but their deployment is constrained by high computational demands. Deploying smaller LLMs directly on the edge can circumvent this, but with degraded accuracy. Deploying smaller cloud-based big LLMs preserves performance, but at the cost of expensive per-token computation. We present a distributed inference framework, \our{}, that integrates speculative decoding (SD) across edge and cloud. A compact draft model deployed on the edge generates candidate tokens rapidly, and a large verifier model on the cloud validates these tokens in parallel. Accepted tokens are retained, while only rejections trigger verifier correction, substantially reducing the number of cloud queries. Our plug-and-play design shifts the bulk of computation to the edge, significantly lowers inference time and cloud cost, and preserves the accuracy of the big model without any retraining requirement. Our approach demonstrates a practical path toward scalable, cost-efficient, and accurate deployment of LLMs in real-world environments. Experimental results across multiple Natural Language Processing tasks using SpecBench and CNN/Dailymail datasets demonstrate that \our{} reduces the cloud model calls by $76\%$ with zero loss in accuracy as compared to the full model.

边缘计算大模型推理推测解码成本优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。