提出连续时间马尔可夫链框架，统一并改进插入式语言模型生成方式。
A Continuous-Time Markov Chain Framework for Insertion Language Models
- 将噪声过程建模为序列空间上的连续时间马尔可夫链，推导出扩散式去噪目标。
- 在合成规划任务中，生成效果优于传统左向生成和掩码扩散模型。
- 采样更灵活，性能媲美主流生成方法，适合需要可控生成的场景。
插入式语言模型（ILMs）相比传统的从左到右生成和基于掩码的生成具有诸多优势。然而，现有插入生成的表述多为临时设计。本文从第一原理出发，将噪声过程建模为变量长度序列空间上的连续时间马尔可夫链，推导出一种扩散风格的去噪目标。我们证明，先前的ILM表述可视为该去噪框架的特例。在合成规划任务上的实证评估表明，所提方法在保留插入生成优势的同时，性能优于左向生成与掩码扩散模型。在语言建模任务中，该扩散方法表现与左向生成和掩码扩散模型相当，同时相较于现有ILMs提供了更高的采样灵活性。
原文摘要 · Abstract (English)
Insertion Language Models (ILMs) offer several advantages over left-to-right generation and mask-based generation. However, existing formulations of insertion-based generation have largely been ad-hoc. In this paper, we derive a diffusion-style denoising objective for ILMs from first principles by formulating the noising process as a continuous-time Markov chain on the space of variable-length sequences. We show that previous formulations of ILMs can be viewed as special cases of this denoising framework. Through empirical evaluation on a synthetic planning task, we show that the proposed approach retains the benefits of insertion-based generation over left-to-right generation and masked diffusion models. In language modeling, our diffusion-based approach is competitive with left-to-right generation and masked diffusion models, while offering additional flexibility in sampling compared to existing insertion language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。