提出可直接在离散时间训练的自校正扩散模型，提升生成效率。
Generalized Discrete Diffusion with Self-Correction
- 基于BERT式目标，在离散时间显式建模状态转移
- 训练噪声调度更简单，无需冗余重掩码步骤
- 在GPT-2规模下实现高效并行生成且保持质量
自校正是一种在离散扩散模型中维持并行采样的有效技术，能以最小性能损失保持生成能力。先前工作多在推理或后训练阶段进行自校正，但普遍存在泛化能力有限、影响推理表现的问题。GIDD首次提出基于预训练的自校正方法，采用多步BERT式均匀吸收目标；然而其依赖连续插值流程，均匀转移与吸收掩码间的交互不透明，导致超参数调优困难，制约实际性能。本文提出自校正离散扩散（SCDD）模型，将预训练自校正重构为显式状态转移机制，直接在离散时间学习。该框架简化了训练噪声调度，移除冗余重掩码步骤，仅依赖均匀转移实现自校正。在GPT-2规模的实验表明，本方法在保持生成质量的同时，实现更高效的并行解码。
原文摘要 · Abstract (English)
Self-correction is an effective technique for maintaining parallel sampling in discrete diffusion models with minimal performance degradation. Prior work has explored self-correction at inference time or during post-training; however, such approaches often suffer from limited generalization and may impair reasoning performance. GIDD pioneers pretraining-based self-correction via a multi-step BERT-style uniform-absorbing objective. However, GIDD relies on a continuous interpolation-based pipeline with opaque interactions between uniform transitions and absorbing masks, which complicates hyperparameter tuning and hinders practical performance. In this work, we propose a Self-Correcting Discrete Diffusion (SCDD) model to reformulate pretrained self-correction with explicit state transitions and learn directly in discrete time. Our framework also simplifies the training noise schedule, eliminates a redundant remasking step, and relies exclusively on uniform transitions to learn self-correction. Experiments at the GPT-2 scale demonstrate that our method enables more efficient parallel decoding while preserving generation quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。