arXiv:2505.22165cs.CLcs.AI2025-05ACL被引 4

新模型统一离散与连续文本扩散,实现更灵活的噪声控制和生成质量提升。

Unifying Continuous and Discrete Text Diffusion with Non-simultaneous Diffusion Processes

  • 采用泊松过程实现非同步噪声添加,支持不同词元独立演化
  • 逆过程引入时间预测器,按语义动态调节去噪进度
  • 在多个任务上超越现有非自回归与自回归方法

扩散模型已成为文本生成的有前景方法,主要分为离散与连续两类。离散模型对词元独立应用类别分布扰动,可实现不同词元的差异化扩散进程,但控制粒度不足;连续模型将词元映射至连续空间并施加精细噪声,但扩散进程对所有词元统一,难以捕捉语义细微差别。为此,本文提出非同步连续扩散模型(NeoDiff),融合两者优势。NeoDiff在前向过程中引入泊松扩散过程,实现灵活且细粒度的噪声注入;在反向过程中采用时间预测器,根据词元语义自适应调节去噪进度。此外,通过优化推理调度,实现更精确的噪声控制并提升性能。该方法统一了离散与连续扩散模型的理论框架,为文本生成提供了更系统有效的范式。在多个文本生成任务上的实验表明,NeoDiff优于非自回归连续与离散扩散模型、基于迭代的方法以及自回归扩散方法,展现出生成高质量文本的潜力,推动了基于扩散的文本生成发展。

原文摘要 · Abstract (English)

Diffusion models have emerged as a promising approach for text generation, with recent works falling into two main categories: discrete and continuous diffusion models. Discrete diffusion models apply token corruption independently using categorical distributions, allowing for different diffusion progress across tokens but lacking fine-grained control. Continuous diffusion models map tokens to continuous spaces and apply fine-grained noise, but the diffusion progress is uniform across tokens, limiting their ability to capture semantic nuances. To address these limitations, we propose \textbf{\underline{N}}on-simultan\textbf{\underline{e}}ous C\textbf{\underline{o}}ntinuous \textbf{\underline{Diff}}usion Models (NeoDiff), a novel diffusion model that integrates the strengths of both discrete and continuous approaches. NeoDiff introduces a Poisson diffusion process for the forward process, enabling a flexible and fine-grained noising paradigm, and employs a time predictor for the reverse process to adaptively modulate the denoising progress based on token semantics. Furthermore, NeoDiff utilizes an optimized schedule for inference to ensure more precise noise control and improved performance. Our approach unifies the theories of discrete and continuous diffusion models, offering a more principled and effective framework for text generation. Experimental results on several text generation tasks demonstrate NeoDiff's superior performance compared to baselines of non-autoregressive continuous and discrete diffusion models, iterative-based methods and autoregressive diffusion-based methods. These results highlight NeoDiff's potential as a powerful tool for generating high-quality text and advancing the field of diffusion-based text generation.

文本生成扩散模型非自回归

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。