将自回归模型转为并行扩散模型,支持两种推理方式且吞吐提升。
FLARE: Diffusion for Hybrid Language Model

- 通过统一目标函数和硬件优化内核,实现混合注意力模型的平滑转换
- 单卡并发服务下吞吐率显著高于开源扩散语言模型基线
- 适合需要低延迟与高并发部署的场景,如实时对话系统
自回归大语言模型虽已广泛应用,但串行解码仍是低延迟部署的关键瓶颈。现有高效推理研究主要沿两个方向:通过高效架构降低每次调用成本,或通过并行生成减少串行步骤。混合注意力架构解决前者,扩散语言模型(dLLMs)则通过迭代并行去噪实现后者。然而两者的结合仍具挑战:自回归到dLLM的转换常导致种子-检查点能力丢失,且混合注意力的递归状态与掩码约束使扩散训练与推理复杂化。本文提出FLARE,一种面向混合注意力模型的系统性转换框架。分析表明,迁移数据质量是能力保留的首要因素,优于损失函数设计与注意力掩码。该框架融合等令牌的自回归与扩散目标、硬件感知内核及统一推理机制,使单一检查点同时支持自回归验证解码与扩散式并行去噪。从少量微调数据的强自回归检查点出发,FLARE在不同模型规模下均具备与领先开源dLLM相当的表现,并在单卡并发服务中实现持续吞吐提升。结果进一步表明,实用dLLM不仅受限于解码算法,还受制于迁移数据质量及当前块扩散目标的训练效率,亟需数据、目标、架构与推理系统的联合设计。
原文摘要 · Abstract (English)
Autoregressive (AR) large language models (LLMs) have achieved broad practical success, but sequential decoding remains a key bottleneck for low-latency deployment. Recent efficient-inference work has progressed along two axes: reducing the cost of each model invocation through efficient architectures, and reducing serial decoding steps through parallel generation. Hybrid attention backbones address the former, while diffusion language models (dLLMs) pursue the latter via iterative parallel denoising. Combining these advantages remains challenging: AR-to-dLLM conversion often fails to preserve seed-checkpoint capability, and hybrid-attention recurrent states and masking constraints make diffusion training and serving nontrivial. We present FLARE, a systematic conversion framework for hybrid-attention LLMs. Our analysis identifies transfer data quality as the primary determinant of capability preservation, outweighing loss formulation and attention-mask design. The resulting framework combines a token-equal AR-and-diffusion objective, hardware-aware kernels, and unified inference, enabling one checkpoint to support both AR-style verified decoding and diffusion-style parallel denoising. Starting from strong AR checkpoints with limited post-training data, FLARE is competitive with leading open-source dLLMs across model scales and delivers consistent throughput gains over open-source dLLM baselines in single-GPU concurrent serving. Our results further suggest that practical dLLMs are limited not only by decoding algorithms, but also by transfer data quality and the training inefficiency of current block-diffusion objectives, motivating joint design of data, objectives, architectures, and inference systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。