不用变压器,用频域混合状态空间实现高效文本生成。
State Fourier Diffusion Language Model (SFDLM): A Scalable, Novel Iterative Approach to Language Modeling
- 用状态空间与傅里叶MLP替代注意力机制,实现无变压器文本生成
- 在WikiText-103上达到与Transformer相当的困惑度(21.7)
- 适合追求低延迟、高可扩展性的生成模型研究者
近年来,基于扩散的方法已成为生成建模的强大范式。尽管离散扩散在自然语言处理中研究较少,但在需要迭代去噪的标记数据任务中展现出潜力。标准文本生成方法依赖变换器,但其自注意力机制常导致高计算成本。本文提出一种完全基于扩散的离散文本生成模型,无需变换器或大型卷积模块。模型在时域结合结构化状态空间动态,引入新型复数傅里叶多层感知机,在频域进行运算。前向加噪过程以可控概率随机替换词汇,而学习到的逆向模型则系统性地将受损序列还原至原始状态。通过局部状态空间更新与全局傅里叶混合的组合,该方法有效捕捉短距离与长距离依赖关系。
原文摘要 · Abstract (English)
In recent years, diffusion based methods have emerged as a powerful paradigm for generative modeling. Although discrete diffusion for natural language processing has been explored to a lesser extent, it shows promise for tasks requiring iterative denoising of token based data. In standard approaches to text generation, transformers dominate, but their reliance on self attention often incurs high computational costs. This paper introduces a fully diffusion driven discrete text generation model built without any transformer or large convolution modules. Instead, the model integrates structured state space dynamics in the time domain with a novel Complex Fourier Multi Layer Perceptron module that operates in the frequency domain. The forward noising process randomly samples the vocabulary to replace tokens with a controlled probability, while the learned reverse model systematically reverts corrupted sequences toward their original states. By composing local state space updates with global Fourier based mixing, the approach effectively captures both short and long range dependencies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。