用扩散模型并行思考,自回归采样说话,实现高速高质生成。
TiDAR: Think in Diffusion, Talk in Autoregression
- 先用扩散模型并行构思文本,再用自回归方式精细生成。
- 在1.5B和8B模型上达到4.71至5.91倍于传统方法的生成速度。
- 首次在保持自回归质量的同时大幅提升吞吐量,适合部署使用。
扩散语言模型具备快速并行生成的潜力,而自回归(AR)模型因因果结构更契合语言建模,通常在质量上表现更优。这引发了一个核心问题:能否兼顾高吞吐、高GPU利用率与自回归级质量?现有方法难以平衡,或依赖弱模型进行串行草稿(推测解码),效率低下;或对扩散模型采用类自回归解码,导致质量下降且丧失并行优势。本文提出TiDAR,一种序列级混合架构:在单次前向传播中,通过特殊设计的结构化注意力掩码,以扩散模型并行生成候选词(思考),再以自回归方式采样最终输出(说话)。该设计充分利用空闲显存计算密度,实现草稿与验证能力的强平衡。此外,TiDAR为服务友好型,可作为独立模型部署。我们在1.5B和8B规模下,对生成与似然任务进行了全面评估。得益于并行草稿与采样以及精确的键值缓存支持,TiDAR在吞吐量上优于推测解码,且在效率与质量上均超越Dream与Llada等扩散模型。最显著的是,TiDAR首次在保持自回归模型质量的同时,实现4.71至5.91倍的每秒生成词数提升。
原文摘要 · Abstract (English)
Diffusion language models hold the promise of fast parallel generation, while autoregressive (AR) models typically excel in quality due to their causal structure aligning naturally with language modeling. This raises a fundamental question: can we achieve a synergy with high throughput, higher GPU utilization, and AR level quality? Existing methods fail to effectively balance these two aspects, either prioritizing AR using a weaker model for sequential drafting (speculative decoding), leading to lower drafting efficiency, or using some form of left-to-right (AR-like) decoding logic for diffusion, which still suffers from quality degradation and forfeits its potential parallelizability. We introduce TiDAR, a sequence-level hybrid architecture that drafts tokens (Thinking) in Diffusion and samples final outputs (Talking) AutoRegressively - all within a single forward pass using specially designed structured attention masks. This design exploits the free GPU compute density, achieving a strong balance between drafting and verification capacity. Moreover, TiDAR is designed to be serving-friendly (low overhead) as a standalone model. We extensively evaluate TiDAR against AR models, speculative decoding, and diffusion variants across generative and likelihood tasks at 1.5B and 8B scales. Thanks to the parallel drafting and sampling as well as exact KV cache support, TiDAR outperforms speculative decoding in measured throughput and surpasses diffusion models like Dream and Llada in both efficiency and quality. Most notably, TiDAR is the first architecture to close the quality gap with AR models while delivering 4.71x to 5.91x more tokens per second.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。