在边缘设备上通过内存感知调度，提升大模型推测解码的生成速度。
MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices
- 用轻量预测器提前判断草案有效性，分离选择与执行过程。
- 在Jetson Orin Nano上平均提速40.7%，接近理论最优性能。
- 适合资源受限的边缘部署场景，尤其关注推理效率的开发者。
推测解码通过使用轻量级草案模型推测多个标记，减少昂贵的目标模型解码步骤，从而加速自回归大语言模型（LLM）的推理。其效果高度依赖于草案选择策略，促使采用适应性方法以利用输入和生成阶段间的差异。然而，在内存受限的边缘设备上，这些方法常因在不同草案模型间切换的开销而无法提升端到端吞吐量。我们发现该场景下的关键限制在于：在严苛内存预算下，草案选择与草案可用性之间的不匹配。为此，我们提出MemSpec，一种基于预测的、内存感知的运行时系统，用于边缘设备上的自适应推测解码。MemSpec通过主动的驻留工作集管理，将草案选择与执行解耦。一个轻量级预测器从提示和生成上下文中估计草案有效性，而一个内存感知调度器则减少反应式模型加载开销。在Jetson Orin Nano上的实验表明，与最先进的基于赌徒算法的自适应方法相比，MemSpec在稳态生成吞吐量上平均提升40.7%，并接近理想基准（oracle upper bound）。
原文摘要 · Abstract (English)
Speculative decoding accelerates autoregressive large language model (LLM) inference by using a lightweight draft model to speculate multiple tokens, reducing expensive target model decoding steps. Its effectiveness depends heavily on draft selection, motivating adaptive methods that exploit variation across inputs and generation stages. On memory-constrained edge devices, however, these methods often fail to improve end-to-end throughput due to the overhead of switching between draft models. We identify a key limitation in this setting: the mismatch between draft selection and draft availability under tight memory budgets. To address this challenge, we present MemSpec, a prediction-guided, memory-aware runtime for adaptive speculative decoding on edge devices. MemSpec decouples draft selection from execution through proactive resident working-set management. A lightweight predictor estimates draft effectiveness from prompt and generation context, while a memory-aware scheduler reduces reactive model loading overhead. Experiments on a Jetson Orin Nano show that MemSpec improves steady-state generation throughput by 40.7% on average over state-of-the-art bandit-based adaptive methods while closely approaching the oracle upper bound.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。