用扩散模型提升文本生成多样性与可控性。
Conditional [MASK] Discrete Diffusion Language Model
- 将掩码语言模型融入扩散框架，构建条件马尔可夫随机场。
- 提出熵自适应采样与噪声调度，平衡生成质量与多样性。
- 适合需要高可控性与多样性的非自回归文本生成任务。
尽管自回归模型在自然语言处理中表现优异，但常面临生成文本多样性不足和可控性差的问题。非自回归方法虽为替代方案，却易产生退化输出，且在条件生成方面存在缺陷。为此，我们提出 Diffusion-EAGS 框架，通过条件马尔可夫随机场的理论视角，将条件掩码语言模型融入扩散语言模型。为此，我们设计了熵自适应吉布斯采样与基于熵的噪声调度策略，以缓解各模型的局限性。实验表明，Diffusion-EAGS 在多个基准上优于基线模型，实现了最佳的质量-多样性权衡，验证了其在非自回归文本生成中的有效性。
原文摘要 · Abstract (English)
Although auto-regressive models excel in natural language processing, they often struggle to generate diverse text and provide limited controllability. Non-auto-regressive methods could be an alternative but often produce degenerate outputs and exhibit shortcomings in conditional generation. To address these challenges, we propose Diffusion-EAGS, a novel framework that integrates conditional masked language models into diffusion language models through the theoretical lens of a conditional Markov Random Field. In doing so, we propose entropy-adaptive Gibbs sampling and entropy-based noise scheduling to counterbalance each model's shortcomings. Experimental results show that Diffusion-EAGS outperforms baselines and achieves the best quality-diversity tradeoff, demonstrating its effectiveness in non-autoregressive text generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。