将自回归与扩散模型结合，实现高效训练和并行生成。
Causal Autoregressive Diffusion Language Model
- 用因果掩码重构扩散过程，单次前向即可全词监督。
- 训练延迟比块扩散低3倍，性能超越现有离散扩散方法。
- 适合追求高吞吐、低延迟的下一代大模型应用。
本文提出一种新型框架CARD，融合自回归模型的训练效率与扩散模型的高吞吐推理优势。CARD在严格因果注意力掩码下重定义扩散过程，实现单次前向传播中的密集逐词监督。为解决因果扩散的优化不稳定性，引入软尾掩码以保留局部上下文，并基于信噪比原理设计上下文感知重加权机制。该设计支持动态并行解码，模型利用KV缓存自适应生成变长词元序列。实验表明，CARD在性能上超越现有离散扩散基线，训练延迟较块扩散降低3倍。结果证明，CARD在保持自回归模型数据效率的同时，实现了并行生成的延迟优势，为下一代高效大模型提供了可靠范式。
原文摘要 · Abstract (English)
In this work, we propose Causal Autoregressive Diffusion (CARD), a novel framework that unifies the training efficiency of ARMs with the high-throughput inference of diffusion models. CARD reformulates the diffusion process within a strictly causal attention mask, enabling dense, per-token supervision in a single forward pass. To address the optimization instability of causal diffusion, we introduce a soft-tailed masking schema to preserve local context and a context-aware reweighting mechanism derived from signal-to-noise principles. This design enables dynamic parallel decoding, where the model leverages KV-caching to adaptively generate variable-length token sequences based on confidence. Empirically, CARD outperforms existing discrete diffusion baselines while reducing training latency by 3 $\times$ compared to block diffusion methods. Our results demonstrate that CARD achieves ARM-level data efficiency while unlocking the latency benefits of parallel generation, establishing a robust paradigm for next-generation efficient LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。