通过平滑词嵌入实现文本生成，兼顾语义与解码自然性。
Smoothie: Smoothing Diffusion on Token Embeddings for Text Generation
- 基于语义相似性逐步平滑词嵌入，融合连续与离散优势。
- 在多个生成任务中超越现有扩散模型，提升生成质量。
- 适合追求高质量文本生成的开发者与研究者使用。
扩散模型在图像、音频和视频生成上已达顶尖水平，但其在文本生成中的应用仍面临挑战，主要源于文本的离散特性。已有方法或在连续潜在空间应用高斯扩散（保留语义结构但解码困难），或在类别单纯形空间操作（保持离散性但忽略词间语义关系）。本文提出一种新方法——词嵌入平滑扩散（Smoothie），通过基于语义相似性的渐进式平滑词嵌入，实现信息逐步消除的同时保持自然解码过程。在多个序列到序列及无条件生成任务上的实验表明，Smoothie 在生成质量上优于现有扩散模型。消融实验进一步证明，该扩散空间性能优于标准嵌入空间和类别单纯形空间。代码已开源：https://github.com/ashaba1in/smoothie。
原文摘要 · Abstract (English)
Diffusion models have achieved state-of-the-art performance in generating images, audio, and video, but their adaptation to text remains challenging due to its discrete nature. Prior approaches either apply Gaussian diffusion in continuous latent spaces, which inherits semantic structure but struggles with token decoding, or operate in categorical simplex space, which respect discreteness but disregard semantic relation between tokens. In this paper, we propose Smoothing Diffusion on Token Embeddings (Smoothie), a novel diffusion method that combines the strengths of both approaches by progressively smoothing token embeddings based on semantic similarity. This technique enables gradual information removal while maintaining a natural decoding process. Experimental results on several sequence-to-sequence and unconditional generation tasks demonstrate that Smoothie outperforms existing diffusion-based models in generation quality. Furthermore, ablation studies show that our proposed diffusion space yields better performance than both the standard embedding space and the categorical simplex. The code is available at https://github.com/ashaba1in/smoothie.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。