通过并行采样非连续未来词元，加速语言模型推理。
Set Block Decoding is a Language Model Inference Accelerator
- 将标准预测与掩码预测融合，实现非连续词元并行生成。
- 在相同精度下，推理次数减少3-5倍，显著提速。
- 无需修改架构，可直接微调现有模型，适合部署优化。
自回归语言模型虽功能强大，但在实际部署中因解码阶段计算与内存开销高而面临挑战。本文提出集块解码（Set Block Decoding, SBD），一种简单灵活的范式，将标准下一个词元预测（NTP）与掩码词元预测（MATP）整合于同一架构中。SBD 允许模型并行采样多个非连续的未来词元，这一特性区别于以往加速方法。该灵活性使我们可引入离散扩散文献中的先进求解器，在不牺牲准确性的前提下实现显著加速。SBD 不需架构改动或额外训练超参数，兼容精确的键值缓存（KV-caching），且可通过微调现有 NTP 模型实现。通过对 Llama-3.1 8B 和 Qwen-3 8B 的微调实验表明，SBD 在保持等效性能的前提下，将生成所需的前向传播次数减少 3-5 倍。
原文摘要 · Abstract (English)
Autoregressive next token prediction language models offer powerful capabilities but face significant challenges in practical deployment due to the high computational and memory costs of inference, particularly during the decoding stage. We introduce Set Block Decoding (SBD), a simple and flexible paradigm that accelerates generation by integrating standard next token prediction (NTP) and masked token prediction (MATP) within a single architecture. SBD allows the model to sample multiple, not necessarily consecutive, future tokens in parallel, a key distinction from previous acceleration methods. This flexibility allows the use of advanced solvers from the discrete diffusion literature, offering significant speedups without sacrificing accuracy. SBD requires no architectural changes or extra training hyperparameters, maintains compatibility with exact KV-caching, and can be implemented by fine-tuning existing next token prediction models. By fine-tuning Llama-3.1 8B and Qwen-3 8B, we demonstrate that SBD enables a 3-5x reduction in the number of forward passes required for generation while achieving same performance as equivalent NTP training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。