用轻量适配器做推测生成,结果算力没省下来。
Accepted Prefixes Are Not All You Need: A Negative Result on PEFT-Based Block-Diffusion Drafting
- 用类似LoRA的适配器作为块扩散采样器,平行生成多词
- 实验显示虽有可接受前缀,但推理速度未提升
- 关键发现:采样器必须远比验证器便宜才有效
推测解码通过廉价采样器预提出未来多个词元,再由目标模型验证来加速自回归语言模型推理。常见目标是提升采样质量的同时降低额外参数与系统开销。我们研究了基于PEFT-BD的方法,这是一种在同一骨干网络上进行推测解码的技术,其中类似LoRA的适配器充当块扩散采样器,用于自回归验证器。该方法具有多项优势:避免分词器不匹配、无需加载独立采样模型、仅增加少量可训练参数,并采用BD3LM风格的去噪目标并行生成词元块。然而,在Qwen3-0.6B实验中,该方法未能带来实际加速。尽管获得了非平凡的接受前缀,性能剖析显示每个推测步骤需执行一次启用适配器的完整骨干网络采样,再执行一次禁用适配器的完整骨干网络验证。因此,虽然参数高效,但计算效率并未提升。结果揭示了成功推测解码的一个简单但关键条件:采样器的执行成本必须远低于验证器。当采样计算仍处于验证器级别时,更长的接受前缀也无法弥补。
原文摘要 · Abstract (English)
Speculative decoding accelerates autoregressive language model inference by using a cheap drafter to propose multiple future tokens and a target model to verify them. A common design goal is therefore to improve draft quality while reducing auxiliary parameters and systems overhead. We study a negative result for this direction through PEFT-BD, a same-backbone speculative decoding method in which a LoRA-like adapter acts as a block-diffusion drafter for an autoregressive verifier. PEFT-BD is motivated by several attractive properties: it avoids tokenizer mismatch, avoids loading a separate draft model, adds only a small number of trainable parameters, and uses a BD3LM-style denoising objective to propose a block of tokens in parallel. Despite these advantages, PEFT-BD does not yield a practical speedup in our Qwen3-0.6B experiments. Although the method obtains nontrivial accepted prefixes, profiling shows that each speculative step requires an adapter-enabled full-backbone draft pass followed by an adapter-disabled full-backbone verification pass. Thus, the drafter is parameter-efficient but not compute-efficient. Our results isolate a simple but important condition for successful speculative decoding: the drafter must be substantially cheaper to execute than the verifier. Longer accepted prefixes alone cannot compensate when draft computation remains verifier-scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。