arXiv:2510.06303cs.LGcs.AI2025-10ACL被引 108

将自回归模型转为可并行生成的扩散模型,兼顾效率与速度。

SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation

论文配图:SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation
图 1 · 摘自论文原文
  • 用轻量转换将训练好的自回归模型变为分块扩散模型。
  • 生成时块间自回归保持连贯性,块内并行解码提升速度。
  • 适合需要高效推理和强泛化能力的大模型场景。

我们提出SDAR,一种融合自回归与扩散模型优势的协同生成范式,兼具自回归模型的训练效率与扩散模型的并行推理能力。不同于昂贵的端到端扩散训练,SDAR通过简短、数据高效的适配,将已训练好的自回归(AR)模型转化为分块扩散模型。推理时,序列在块间自回归生成以保证全局连贯性,同时在每个块内通过离散扩散过程并行解码所有标记。大量实验表明,自回归模型在计算效率上仍显著优于掩码扩散模型,为转化提供了坚实基础。基于此,SDAR以极低代价实现从AR到扩散的转化,保持自回归水平性能的同时支持并行生成。在密集型与专家混合(Mixture-of-Experts)架构上的扩展研究证实,SDAR可无损扩展:模型越大,对块大小和解码阈值越鲁棒,带来更大加速且不损失准确率。此外,SDAR展现出更强的推理能力和领域适应性。我们的300亿参数MoE模型在GPQA和ChemBench等复杂科学推理基准上超越其自回归版本,并在测试时缩放方法如多数投票和pass@k下进一步提升。这些结果确立了SDAR作为可扩展、高吞吐推理的实用范式。

原文摘要 · Abstract (English)

We propose SDAR, a Synergistic Diffusion-Autoregression paradigm that unifies the training efficiency of autoregressive models with the parallel inference capability of diffusion. Instead of costly end-to-end diffusion training, SDAR performs a lightweight paradigm conversion that transforms a well-trained autoregressive (AR) model into a blockwise diffusion model through brief, data-efficient adaptation. During inference, SDAR generates sequences autoregressively across blocks for global coherence while decoding all tokens within each block in parallel via a discrete diffusion process. Extensive experiments show that AR models remain substantially more compute-efficient than masked diffusion models, providing a strong foundation for adaptation. Building on this insight, SDAR achieves efficient AR-to-diffusion conversion with minimal cost, preserving AR-level performance while enabling parallel generation. Scaling studies across dense and Mixture-of-Experts architectures confirm that SDAR scales without compromise: larger models exhibit stronger robustness to block size and decoding thresholds, yielding greater speedups without accuracy loss. Beyond efficiency, SDAR demonstrates enhanced reasoning and domain adaptability. Our 30B MoE model surpasses its AR counterpart on challenging scientific reasoning benchmarks such as GPQA and ChemBench, and gains further improvements under test-time scaling methods like majority voting and pass@k. Together, these results establish SDAR as a practical paradigm that combines the strengths of autoregression and diffusion for scalable, high-throughput reasoning.

扩散模型自回归大模型推理并行生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。