通过预定位技术让大模型提前准备好下一决策点，显著提速响应。
Speculative Pre-Positioning: Decoding Stateful Sessions to the Next Decision Point Off the Critical Path
- 用目标模型自身前向计算预推进会话到下一决策点。
- 在高置信度下可1毫秒内返回首个词元，较缓存快39倍。
- 适合追求低延迟的生成服务，尤其大模型场景。
无状态推理服务器（如vLLM、SGLang、TensorRT-LLM）在请求间处于空闲状态，而有状态会话可利用这段空闲时间。推测性预定位使用目标模型自身的前向传播，将会话推进至下一个决策点，无需草稿模型，从而将跨请求预填充和进入解码移出关键路径：下一请求可从已预付的入口继续，或当置信度门限触发时，以近乎恒定的词汇表扫描一次性返回缓存分布，无需解码，仅消耗能量和少量可控的误接受。该方法收益取决于模型能力：能力强的模型在接近全覆盖率下达到约87%精度（小模型无法通过），首个词元可在约1.0毫秒内返回，相较前缀缓存仍需39毫秒解码的方案大幅提升性能。
原文摘要 · Abstract (English)
A stateless inference server (vLLM, SGLang, TensorRT-LLM) idles between requests while the accelerator waits; a stateful session reclaims that idle time. Speculative pre-positioning decodes the session forward to its next decision point with the target model's own forward pass and no draft model, moving the cross-request prefill and entry-decode off the critical path: the next request resumes from a pre-paid entry on its delta, or, when a confidence gate fires, is answered from a cached distribution in one near-constant vocabulary scan with no decode, at a cost only of energy and a rare, bounded false accept. The payoff is conditional on capability: a capable model fires the gate at near-full coverage and about 87% precision (a smaller one never clears it), returning the first token in about 1.0 ms versus the 39 ms decode a prefix cache still pays.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。