让离散扩散模型学会回看历史，提升生成质量
Non-Markovian Discrete Diffusion with Causal Language Models
- 引入非马尔可夫机制，允许模型回顾整个生成过程
- 在自然语言任务上超越现有离散扩散模型，接近自回归大模型表现
- 可直接复用预训练语言模型权重，无需修改架构
离散扩散模型在结构化序列生成中具有灵活性和可控性，但其表达能力仍落后于因果语言模型。核心限制在于依赖马尔可夫假设，即每一步仅基于当前状态，导致错误累积难以纠正。本文提出CaDDi（因果离散扩散模型），通过条件于完整生成轨迹，打破马尔可夫约束，使模型能够回溯并修正过去状态。通过在单一非马尔可夫Transformer中统一序列（因果）与时间（扩散）推理，CaDDi将标准因果语言模型视为特例，并支持直接复用预训练LLM权重而无需架构改动。实证表明，CaDDi在自然语言基准上优于现有先进离散扩散基线，显著缩小与大型自回归变换器的差距。
原文摘要 · Abstract (English)
Discrete diffusion models offer a flexible, controllable approach to structured sequence generation, yet they still lag behind causal language models in expressive power. A key limitation lies in their reliance on the Markovian assumption, which restricts each step to condition only on the current state, leading to potential uncorrectable error accumulation. In this paper, we introduce CaDDi (Causal Discrete Diffusion Model), a discrete diffusion model that conditions on the entire generative trajectory, thereby lifting the Markov constraint and allowing the model to revisit and improve past states. By unifying sequential (causal) and temporal (diffusion) reasoning in a single non-Markovian transformer, CaDDi also treats standard causal language models as a special case and permits the direct reuse of pretrained LLM weights with no architectural changes. Empirically, CaDDi outperforms state-of-the-art discrete diffusion baselines on natural-language benchmarks, substantially narrowing the remaining gap to large autoregressive transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。