通过分层路由提升大模型推理效率，减少冗余计算。
VIA-SD: Verification via Intra-Model Routing for Speculative Decoding
- 引入多级验证机制，按置信度分治处理候选词
- 在多个任务上将拒绝率降低0.10-0.22，速度提升10%-20%
- 兼容现有框架，无需重新训练，适合部署优化场景
推测解码（SD）通过轻量级草稿模型生成候选序列，由大型验证模型并行验证，以降低大语言模型的推理开销。现有方法采用二元决策：接受或完全重算。我们发现，许多被拒绝的标记可通过从完整验证模型中提取的微型子模型正确验证，而非依赖全模型。为此，我们提出基于模型内路由的推测解码框架VIA-SD，采用分层式轻量验证器。草稿标记按置信度分三类处理：高置信度直接接受，中置信度由轻量验证器重构，低置信度则交由全模型验证。在四个代表性任务及多种模型家族上，VIA-SD将拒绝率降低0.10–0.22，相较强基线提速10%–20%，较非草稿解码加速2.5–3倍。该方法兼容现有SD框架，无需修改训练流程。结果表明，多级推测解码是可扩展高效的大模型推理通用范式。
原文摘要 · Abstract (English)
Speculative decoding (SD) addresses the high inference costs of LLMs by having lightweight drafters generate candidates for large verifiers to validate in parallel. Existing draft-verify methods use binary decisions: accept or fully recompute. Yet we find that many rejected tokens can be verified correctly by a slim submodel derived from the full verifier via intra-model routing, instead of the full verifier. This motivates our slim-verifier to handle tokens requiring moderate verification resources, reducing expensive large-model calls. We propose Verification via Intra-Model Routing for Speculative Decoding (VIA-SD), a multi-tier framework using a routed slim-verifier. Draft tokens are processed hierarchically: direct acceptance for high-confidence cases, slim-verifier regeneration for medium-confidence cases, and full-model verification for uncertain cases. Across four representative tasks and multiple model families, VIA-SD reduces rejection rates by 0.10-0.22 and delivers 10-20% speedups over strong SD baselines, while achieving 2.5-3x acceleration over non-drafting decoding. Moreover, VIA-SD is compatible with existing SD frameworks without modifying their training procedures. Our results suggest multi-tier SD as a general paradigm for scalable and efficient LLM inference. Project page: https://zju-xyc.github.io/VIA-SD-Project-Page/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。