TIDE让大模型推理自适应提速，动态调整猜测生成策略。
TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference
- 将推理过程中的隐藏状态用作训练信号，实现零开销的动态猜测优化。
- 在真实负载下，推理吞吐最高提升1.15倍，训练时间减少1.67倍。
- 适合需要持续优化推理性能的部署场景，尤其适用于异构计算集群。
推测解码可显著加速大模型推理，但实际应用受工作负载变化和系统约束影响。本文提出TIDE（Temporal Incremental Draft Engine），一个原生集成于服务引擎的框架，将在线猜测适配直接融入高性能大模型推理系统。TIDE复用推理过程中目标模型生成的隐藏状态作为训练信号，实现无需重载目标模型的零开销猜测适应，并采用自适应运行时控制，在有益时才激活推测与训练。TIDE通过将解耦的推理与训练映射到不同类别的GPU，有效利用异构集群。在多种真实工作负载下，TIDE相比静态推测解码实现最高1.15倍的吞吐提升，同时比需重新计算训练信号的方法减少1.67倍的猜测训练时间。
原文摘要 · Abstract (English)
Speculative decoding can substantially accelerate LLM inference, but realizing its benefits in practice is challenging due to evolving workloads and system-level constraints. We present TIDE (Temporal Incremental Draft Engine), a serving-engine-native framework that integrates online draft adaptation directly into high-performance LLM inference systems. TIDE reuses target model hidden states generated during inference as training signals, enabling zero-overhead draft adaptation without reloading the target model, and employs adaptive runtime control to activate speculation and training only when beneficial. TIDE exploits heterogeneous clusters by mapping decoupled inference and training to appropriate GPU classes. Across diverse real-world workloads, TIDE achieves up to 1.15x throughput improvement over static speculative decoding while reducing draft training time by 1.67x compared to approaches that recompute training signals.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。