arXiv:2604.26985cs.LGcs.AI2026-04被引 3

让掩码扩散模型在生成时自参考,显著提升文本与分子等序列生成质量。

Simple Self-Conditioning Adaptation for Masked Diffusion Models

  • 用自身前一步的预测结果指导后续去噪,实现跨步信息复用。
  • 在OWT数据集上生成困惑度从42.89降至23.72,降幅近50%。
  • 无需额外训练或模型结构改动,适合快速部署到现有掩码扩散模型。

掩码扩散模型(MDMs)通过迭代去噪生成离散序列,其标准流程中若某位置在反向更新后仍被遮蔽,则丢弃该位置的原始状态预测。因此,仍被遮蔽的位置只能依赖掩码标记反复推断,限制了跨步优化能力。本文提出一种简单有效的后训练适配方法——自条件掩码扩散模型(SCMDM),使每一步去噪均基于模型自身的先前干净状态预测。该方法仅需微小架构调整,不引入循环隐状态路径,也不依赖辅助模型或采样阶段额外去噪器评估。相比需要从头训练的部分自条件化方法,本文发现此类策略在后训练阶段表现不佳;一旦模型自生成的预测具备信息量,专注精细化修正比混合条件与无条件目标更优。SCMDM在多个领域验证有效,相较基线模型显著提升性能:在基于OWT训练的模型上,生成困惑度从42.89降至23.72(降幅近50%),同时在离散图像生成、小分子生成及基因组分布建模方面也表现出更强保真度。

原文摘要 · Abstract (English)

Masked diffusion models (MDMs) generate discrete sequences by iterative denoising under an absorbing masking process. In standard masked diffusion, if a token remains masked after a reverse update, the model discards its clean-state prediction for that position. Thus, still-masked positions must be repeatedly inferred from the mask token alone. This design choice limits cross-step refinement. To address this limitation, this paper proposes a simple, yet effective, post-training adaptation for MDMs that conditions each denoising step on the model's own previous clean-state predictions. The resulting method, called Self-Conditioned Masked Diffusion Models (SCMDM), requires minimal architectural change, does not introduce a recurrent latent-state pathway, does not rely on an auxiliary reference model, and adds no extra denoiser evaluations during sampling. This is an important departure from partial self-conditioning approaches which requires expensive model training from scratch. In particular, the paper shows that partial self-conditioning, including the commonly used 50% dropout strategy for training self-conditioned models from scratch, is suboptimal in the post-training regime. Instead, once the model's self-generated clean-state estimates become informative, the specialization to refinement is preferable to mixing conditional and unconditional objectives. SCMDM is evaluated across multiple domains, demonstrating consistent improvement over vanilla MDM baselines, achieving nearly a 50% reduction in generative perplexity on OWT-trained models (42.89 to 23.72), alongside strong improvements in discretized image synthesis quality, small molecular generation, and enhanced fidelity in genomic distribution modeling.

扩散模型序列生成自条件掩码建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。