首个实现灵活长度文本填充的离散扩散模型,解决位置信息缺失难题。
Flexible-length Text Infilling for Discrete Diffusion Models
- 联合去噪词元值与位置,用最优传输耦合保持顺序动态调整长度。
- 在One-Billion-Word和Yelp数据集上超越基线,性能媲美顶尖非自回归模型。
- 兼容多种预训练文本去噪器,提升训练效率与生成灵活性,适合需要可控填充场景。
离散扩散模型作为新一代文本生成方法,相比自回归模型具备双向上下文利用、并行生成和灵活提示等优势。然而,其关键局限在于无法在无真实位置信息情况下实现灵活长度或位置的文本填充。本文提出首个克服该问题的离散扩散模型DDOT(Discrete Diffusion with Optimal Transport Position Coupling),通过新颖的样本级最优传输(OT)耦合机制,联合去噪词元值与位置,保持词元相对顺序的同时动态调整填充段的位置与长度,这一能力此前在文本扩散模型中尚未实现。该方法与现有离散文本扩散方法正交,可兼容多种预训练文本去噪器。大量实验表明,DDOT在One-Billion-Word和Yelp等文本填充基准上显著优于朴素扩散基线,并达到与当前最先进非自回归模型相当的性能,同时带来显著的训练效率和灵活性提升。
原文摘要 · Abstract (English)
Discrete diffusion models are a new class of text generators that offer advantages such as bidirectional context use, parallelizable generation, and flexible prompting compared to autoregressive models. However, a critical limitation of discrete diffusion models is their inability to perform flexible-length or flexible-position text infilling without access to ground-truth positional data. We introduce \textbf{DDOT} (\textbf{D}iscrete \textbf{D}iffusion with \textbf{O}ptimal \textbf{T}ransport Position Coupling), the first discrete diffusion model to overcome this challenge. DDOT jointly denoises token values and token positions, employing a novel sample-level Optimal Transport (OT) coupling. This coupling preserves relative token ordering while dynamically adjusting the positions and length of infilled segments, a capability previously missing in text diffusion. Our method is orthogonal to existing discrete text diffusion methods and is compatible with various pretrained text denoisers. Extensive experiments on text infilling benchmarks such as One-Billion-Word and Yelp demonstrate that DDOT outperforms naive diffusion baselines. Furthermore, DDOT achieves performance on par with state-of-the-art non-autoregressive models and enables significant improvements in training efficiency and flexibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。