连续扩散模型让文本生成更快更准，支持不同词以不同速度生成。
Token Time Continuous Diffusion for Language Modeling
- 在连续空间中直接映射噪声为文本，无需反复采样。
- 不同词可按需加速生成，高加速下生成质量优于传统模型。
- 适合追求高速生成的场景，尤其擅长条件生成任务。
本文提出一种新型扩散语言模型——令牌时间连续扩散（TTCD），其核心创新在于：(a) 在连续空间中运行，能确定性地将高斯噪声映射为最终文本，无需额外采样；(b) 引入每令牌的时间概念，使部分令牌可比其他令牌更快从噪声演化为真实词。连续空间建模避免了纯离散模型在高速生成时因并行采样导致的精度损失。每令牌时间机制有助于更好地建模条件生成，允许更确定的词加速生成，并在优化过程中实现差异化词间影响。我们在OpenWebText数据集上训练了一个160M参数的TTCD模型，并进行自蒸馏。在高加速条件下，其无条件生成质量与同类模型相当，但在条件生成任务上表现更优，优于相同规模、同数据集训练且自蒸馏的多个现有模型。在数独求解任务中也取得了类似提升。
原文摘要 · Abstract (English)
In this paper we introduce token time continuous diffusion (TTCD), a new diffusion language model which (a) operates in continuous space, deterministically mapping Gaussian noise to a final token canvas with no further sampling, and crucially (b) incorporates a new notion of per-token times, with some tokens proceeding from noise to token at a faster rate than others. Continuous space modeling helps TTCD avoid the parallel sampling of multiple tokens, which is a key source of inaccuracy at high speedups for models that iterate purely in discrete space. The notion of per-token times helps TTCD to better model conditional generation, allows for more sure tokens to proceed at a faster rate, and allows for differentiated inter-token influences during refinement. TTCD outperforms discrete models at high speedups. We train a 160M parameter TTCD model on OpenWebText, and then self-distill it; we find that at high speedups we are comparable in unconditional generation quality, and outperform in conditional generation, several existing models of similar size trained, on the same data, and self-distilled. We achieve similar gains in Sudoku solving as well.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。