动态调整模型链路,让大模型推理更快更省资源。
SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Models
- 根据实时性能自动选配不同大小的模型组合
- 多级模型协同验证,减少最终模型负担
- 支持异步批处理的高效缓存管理,适合复杂请求
大型语言模型在推理质量与计算成本间存在关键权衡:更大模型能力更强但延迟高,更小模型快但弱。现有服务策略多采用固定模型规模或静态两阶段推测解码,无法动态适应用户请求复杂度或系统性能波动。本文提出SpecRouter,将大模型推理重构为自适应路由问题,通过多层级推测解码实现动态路径构建与优化。核心贡献包括:(1) 自适应模型链调度机制,结合执行时延性能分析与基于词元分布差异的预测相似性,持续选择最优的草稿与验证模型序列,最小化每生成一个词元的预测延迟;(2) 多层级协同验证框架,允许链中中间模型验证推测词元,降低对最终强模型的验证压力;(3) 同步状态管理机制,实现异构模型链中高效的KV缓存一致处理,包含针对异步批处理特性的精确低开销回滚。初步实验验证了方法有效性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) present a critical trade-off between inference quality and computational cost: larger models offer superior capabilities but incur significant latency, while smaller models are faster but less powerful. Existing serving strategies often employ fixed model scales or static two-stage speculative decoding, failing to dynamically adapt to the varying complexities of user requests or fluctuations in system performance. This paper introduces \systemname{}, a novel framework that reimagines LLM inference as an adaptive routing problem solved through multi-level speculative decoding. \systemname{} dynamically constructs and optimizes inference "paths" (chains of models) based on real-time feedback, addressing the limitations of static approaches. Our contributions are threefold: (1) An \textbf{adaptive model chain scheduling} mechanism that leverages performance profiling (execution times) and predictive similarity metrics (derived from token distribution divergence) to continuously select the optimal sequence of draft and verifier models, minimizing predicted latency per generated token. (2) A \textbf{multi-level collaborative verification} framework where intermediate models within the selected chain can validate speculative tokens, reducing the verification burden on the final, most powerful target model. (3) A \textbf{synchronized state management} system providing efficient, consistent KV cache handling across heterogeneous models in the chain, including precise, low-overhead rollbacks tailored for asynchronous batch processing inherent in multi-level speculation. Preliminary experiments demonstrate the validity of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。