通过构建异构树结构,让高可信候选词深度延伸,低可信词广度发散,实现无训练的推理加速。
Goose: Anisotropic Speculation Trees for Training-Free Speculative Decoding

- 利用上下文匹配与统计预测的差异性,构建深干枝+广分支的异构树结构
- 在五种模型上实现1.9-4.3倍无损加速,比均衡树快12-33%
- 适合追求高效推理且无训练成本的部署场景
推测解码通过起草多个候选词并在单次前向传播中验证来加速大语言模型推理。候选词组织成树结构:树越深,每步接受的词越多,但增加深度会牺牲宽度(回退选项),受限于固定验证预算。现有无训练方法从单一词源起草,未区分不同来源候选词的质量。我们观察到两种常见无训练词源——从输入上下文复制的n-gram匹配和先前前向传播的统计预测——在接纳率上存在显著差距(中位数差距约6倍,跨五模型五基准范围2-18倍)。我们证明当存在此类质量差距时,最优树结构为各向异性(非对称):可靠词应形成深链,不可靠词则在每个节点展开为宽分支,从而提升平衡树的深度上限。我们在GOOSE中实现这一结构:一个由高接纳率上下文匹配词构成的自适应主干树,每个节点附带低接纳率替代项的宽分支。该结构在相同预算下至少能接受与任一词源单独使用同等数量的词。在五个大模型(7B-33B)和五个基准上,GOOSE实现了1.9-4.3倍无损加速,优于平衡树基线12-33%。
原文摘要 · Abstract (English)
Speculative decoding accelerates large language model inference by drafting multiple candidate tokens and verifying them in a single forward pass. Candidates are organized as a tree: deeper trees accept more tokens per step, but adding depth requires sacrificing breadth (fallback options) under a fixed verification budget. Existing training-free methods draft from a single token source and shape their trees without distinguishing candidate quality across origins. We observe that two common training-free token sources -- n-gram matches copied from the input context, and statistical predictions from prior forward passes -- differ sharply in acceptance rate (~6x median gap, range 2-18x across five models and five benchmarks). We prove that when such a quality gap exists, the optimal tree is anisotropic (asymmetric): reliable tokens should form a deep chain while unreliable tokens spread as wide branches, raising the depth ceiling of balanced trees. We realize this structure in GOOSE, a training-free framework that builds an adaptive spine tree: a deep chain of high-acceptance context-matched tokens with wide branches of low-acceptance alternatives at each node. The resulting tree provably accepts at least as many tokens per step as either source alone. On five LLMs (7B-33B) and five benchmarks, GOOSE achieves 1.9-4.3x lossless speedup, outperforming balanced-tree baselines by 12-33% under the same budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。