动态调整生成长度，让边缘云大模型推理更快更省时。
Delay-Adaptive Speculation Control for Low-Latency Edge-Cloud LLM Inference
- 根据网络延迟自适应调节猜测生成的长度，平衡速度与准确性。
- 实测在83~111毫秒延迟下出现性能转折点，最优长度随延迟对数增长。
- 适用于网络波动场景，比传统方法低14%以上延迟，适合实时系统部署。
推测解码通过轻量级草稿模型预估词元，由大型目标模型并行验证以加速大语言模型推理。在分布式边缘-云推理中，草稿长度需在线控制：更长的草稿可分摊通信延迟但降低词元接受率，较短草稿虽保持高接受率却引发更多通信轮次。本文将此权衡建模为比例型最优停止问题，并证明最优草稿长度为有限延迟单调阈值。分析揭示，在低于某一临界延迟时，单词元推测最优，且最优长度仅随通信延迟对数增长。针对时变网络，扩展模型至马尔可夫调制信道，在有界时域与单调停止区域条件下，建立状态依赖阈值策略。对于未知环境，提出UCB-SpecStop算法，具备无间隙和带间隙的期望遗憾界 $O(L_{ ext{max}} oot{2}{K_{ ext{max}}T ext{log}(K_{ ext{max}}T)})$ 和 $O(\sum_{k:Δ_k>0}L_{\text{max}}^2\text{log}(K_{\text{max}}T)/Δ_k)$。在真实测试平台上（Jetson Orin Nano Super边缘节点 + RTX 3090 Ti云节点），使用Qwen与Llama作为草稿-目标对进行实验，验证了预测的相变现象，转折点位于83~111毫秒之间。Qwen符合几何预测，而Llama因重头接受特性需经验前缀校准。在测试延迟范围内，UCB-SpecStop相比SpecDec++最多降低22.4%每词元延迟，在通信主导场景下接近离线最优解（差距0.2–2.4%），较朴素UCB提升最高7.5%，消除静态调优在延迟漂移下造成的14.0–18.7%性能差距，并借助上下文信道状态信息获得3.0–6.8%增益。
原文摘要 · Abstract (English)
Speculative decoding accelerates large language model (LLM) inference by using a lightweight draft model to propose tokens and a larger target model to verify them in parallel. In distributed edge-cloud inference, however, draft length must be controlled online: longer drafts amortize communication delay but reduce token acceptance, whereas shorter drafts preserve acceptance but trigger more communication rounds. We formulate this tradeoff as a ratio-type optimal stopping problem and prove that the optimal draft length is a finite delay-monotone threshold. The analysis identifies a critical delay below which single-token speculation is optimal and shows that the optimal length grows only logarithmically with communication delay. For time-varying networks, we extend the model to Markov-modulated channels and establish, under a bounded horizon and monotone stopping-region conditions, a state-dependent threshold policy. For unknown environments, we propose UCB-SpecStop, an online control algorithm with gap-free and gap-dependent expected regret bounds of $O(L_{\max}\sqrt{K_{\max}T\log(K_{\max}T)})$ and $O(\sum_{k:Δ_k>0}L_{\max}^2\log(K_{\max}T)/Δ_k)$. We implement the method on a real edge-cloud testbed with a Jetson Orin Nano Super edge node and an RTX~3090 Ti cloud node, using Qwen and Llama draft--target pairs. Experiments validate the predicted phase transition, with transition points near 83~ms and 111~ms. Qwen matches the geometric prediction, while Llama requires empirical-prefix calibration due to heavy-head acceptance. Across the tested delay grid, UCB-SpecStop reduces per-token latency over SpecDec++ by up to 22.4\%, approaches an offline oracle within 0.2--2.4\% in communication-dominated regimes, improves over naive UCB by up to 7.5\%, removes the 14.0--18.7\% gap caused by static tuning under delay drift, and gains 3.0--6.8\% with contextual channel-state information.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。