让推理更省时:根据成本动态调整生成树大小
Cost-Aware Diffusion Draft Trees for Speculative Decoding

- 基于验证成本优化生成树结构和节点预算,直接提升每秒生成字数
- 在八个任务上表现优于或媲美使用最优预算的基线方法
- 无需预先调参,可随每轮情况自动调节生成规模
推测解码通过轻量级草稿模型并行提出候选词,由目标语言模型验证以加速推理。现有方法如DDTree利用扩散模型生成候选词分布,构建最大化预期接受长度的树结构,但其仅关注接受长度,忽略验证开销,导致始终偏好更大树结构,缺乏预算选择的合理依据。本文提出CaDDTree(成本感知扩散草稿树),显式建模草稿与验证延迟,将吞吐量(单位时间内生成的平均词数)作为目标函数,证明在验证成本凸函数假设下,吞吐量函数为单峰,支持高效贪心停止规则。该方法无需离线搜索最优预算,能根据当前每位置分布和验证成本自适应调整每轮预算。在Qwen3-4B和Qwen3-8B模型上,覆盖推理、编程和指令遵循等八项任务的实验表明,CaDDTree在几乎所有任务中均达到或超过使用已知最优预算的基线性能。
原文摘要 · Abstract (English)
Speculative decoding accelerates inference by having a lightweight drafter propose tokens verified in parallel by the target language model. Block diffusion drafters such as DFlash generate an entire draft block in one pass, yielding per-position marginals; DDTree uses these to build a candidate tree that maximizes expected acceptance length under a fixed node budget. We observe, however, that acceptance length is non-decreasing in budget: it always favors larger trees regardless of verification cost, offering no principled basis for budget selection. We introduce \textbf{CaDDTree} (Cost-aware Diffusion Draft Tree), a method that directly optimizes token throughput (expected tokens generated per unit time) by jointly selecting the tree structure and node budget. We model draft and verification latencies explicitly, show that the throughput objective decomposes into a per-round one-dimensional search over the budget, and prove that under a convex verification cost the throughput function is \emph{unimodal}, enabling an efficient greedy stopping rule. CaDDTree requires no offline budget search, adapting the budget each round from the current per-position distributions and verification cost. Experiments on Qwen3-4B and Qwen3-8B across eight benchmarks spanning reasoning, coding, and instruction-following tasks show that \caDDTree{} matches or surpasses DDTree with oracle budget selection on nearly all tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。