研究不同任务对大模型推测解码接受率的影响，发现聊天任务最易通过验证。
Acceptance Dynamics Across Cognitive Domains in Speculative Decoding
- 用小模型生成候选词树，大模型批量验证，分析四类任务的接受规律
- 聊天任务平均每步接受超1个词，代码与推理任务接受率低且依赖深度
- 意外发现高熵（混乱）的聊天任务反而接受率最高，因符合训练语料风格
推测解码可加速大语言模型推理，通过小模型生成未来词的树结构，再由大模型在单次批处理前向中验证。尽管相关研究众多，但任务的认知特性如何影响接受概率仍不清楚。本文对基于树结构的推测解码接受动态进行了实证研究，涵盖代码生成、数学推理、逻辑推理和开放对话四个经典NLP领域。实验使用TinyLlama-1.1B作为草稿模型，对比Llama-2-7B-Chat-GPTQ作为目标模型，在200个提示下收集超过99,768个推测节点，分析各领域的接受率、期望接受长度、深度-接受分布及熵-接受相关性。结果表明，任务类型比树深度更能预测接受概率；仅聊天领域在每步中期望接受长度超过1.0个词。熵-接受相关性在所有领域均呈负向但微弱（rho ∈ [-0.20, -0.15]）。令人意外的是，聊天任务熵值最高，接受率也最高，我们归因于其符合经过强化学习对齐的语料风格。这些发现对领域感知的推测预算分配和草稿模型选择具有直接指导意义。
原文摘要 · Abstract (English)
Speculative decoding accelerates large language model (LLM) inference. It uses a small draft model to propose a tree of future tokens. A larger target model then verifies these tokens in a single batched forward pass. Despite the growing body of work on speculative methods, the degree to which the cognitive characteristics of a task affect acceptance probability remains largely unexplored. We present an empirical study of tree-based speculative decoding acceptance dynamics. Our study spans four well-established NLP benchmark domains: code generation, mathematical reasoning, logical reasoning, and open-ended chat. For this, we use TinyLlama-1.1B as the draft model against Llama-2-7B-Chat-GPTQ as the target. Over 99,768 speculative nodes collected from 200 prompts, we derive per-domain acceptance rates, expected accepted lengths, depth-acceptance profiles, and entropy-acceptance correlations. We find that task type is a stronger predictor of acceptance than tree depth. Furthermore, only the chat domain consistently yields an expected accepted length exceeding 1.0 token per step. We also show that the entropy-acceptance correlation is consistently negative but weak across all domains (rho in [-0.20, -0.15]). Counterintuitively, chat produces the highest entropy yet the highest acceptance rate. We attribute this divergence to the lexical predictability of RLHF-aligned register. These findings have direct implications for domain-aware speculation budgets and draft-model selection strategies. Index Terms--speculative decoding, large language model inference, tree attention, draft model, acceptance probability, LLM efficiency
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。