Sangam优化扩散语言模型的高效推理，解决预填充与解码冲突问题。
Sangam: Efficiently Serving Diffusion LLMs with the AR Stack
- 引入缺陷令牌预算调度，实现无阻塞的流水线调度。
- 混合部署策略下，预填充溢出到解码节点，降低延迟8-20%。
- 适用于解码密集或预填充密集场景，显著提升服务效率。
扩散语言模型（dLLMs）通过迭代去噪生成文本，每次调用可提交多个输出位置。其双向注意力机制阻止了传统自回归式键值缓存，因任一位置的确定都会改变所有其他位置的键值激活。现有近似缓存技术如Fast-dLLM和dKV-Cache需重复刷新并复用键值，形成反复的预填充/解码结构，使自回归服务机制难以直接适用。dLLM解码以块为单位而非单个标记，预填充周期性出现，且双向注意力禁止分块预填充机制。我们提出Sangam，一种面向缓存式dLLM推理的服务系统。Sangam采用缺陷令牌预算调度器：优先接纳进行中的解码，仅当累积预算足够时才允许完整预填充，并将未使用预算传递至下一周期，实现摊销后的无阻塞调度。分离式部署虽避免预填充与解码干扰，但面临资源划分难题。Sangam采用混合策略，将预填充溢出至解码工作节点，缓解预填充资源不足；同时使用相同调度器保护这些节点的解码不受溢出影响。实验表明，如同自回归服务，dLLM服务设计受预填充-解码干扰与资源划分双重制约。共置部署在解码密集型负载下更优，相比混合执行，使LLaDA-8B ShareGPT平均延迟降低9-20%；而混合执行在预填充密集型负载下更优，在Dream-7B arXiv上相较共置执行降低8-20%。Sangam已开源：https://github.com/UT-InfraAI/sangam。
原文摘要 · Abstract (English)
Diffusion language models (dLLMs) generate text by iteratively denoising a masked response and can commit multiple output positions per model invocation. Their bidirectional attention prevents exact autoregressive-style KV caching, since committing one position shifts the KV activations of all others. Approximate caching techniques such as Fast-dLLM and dKV-Cache refresh KV activations repeatedly and reuse them across intervening decodes, inducing a repeated prefill/decode structure. This makes AR serving mechanisms relevant to dLLMs, but not directly applicable. dLLM decodes are block-sized rather than token-sized, prefills recur, and bidirectional attention precludes the chunked prefill mechanism used for stall-free colocated serving. We present Sangam, a serving system for cached dLLM inference. Sangam introduces a deficit token-budget scheduler that admits in-flight decodes first, admits whole indivisible prefills only when the accumulated token budget allows, and carries unused budget forward. This achieves amortized stall-free scheduling. Disaggregated serving avoids prefill-decode interference but suffers from prefill/decode resource partitioning problem. Sangam adopts a hybrid serving strategy, overflowing prefills onto decode workers to relieve prefill under-provisioning, and uses the same deficit-budget scheduler to protect those workers' decodes from the overflow. We show that like AR serving, dLLM serving design space is governed by prefill-decode interference and prefill/decode partitioning. Colocated serving is most effective on decode-heavy workloads, cutting mean latency by 9-20% over hybrid execution on LLaDA-8B ShareGPT; while hybrid execution is most effective on prefill-heavy workloads, cutting mean latency by 8-20% over colocated execution on Dream-7B arXiv. Sangam is available at https://github.com/UT-InfraAI/sangam.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。