用傅里叶变换实现并行文本生成，速度更快且更连贯。
FourierNAT: A Fourier-Mixing-Based Non-Autoregressive Transformer for Parallel Sequence Generation
- 在解码器中引入傅里叶混合，通过频域操作并行处理序列
- 在WMT和CNN/DailyMail上表现媲美主流非自回归模型，推理速度显著提升
- 可自适应聚焦长程或短程依赖，缓解一次性生成的连贯性缺陷
我们提出FourierNAT，一种基于傅里叶混合的非自回归Transformer架构，通过离散傅里叶变换在解码器中对整个序列维度的词元嵌入进行混合，并结合可学习的频域门控机制，实现输出序列的并行生成。传统非自回归方法常难以捕捉全局依赖，而本方法无需显式自回归步骤即可高效传播上下文。实验表明，FourierNAT在标准基准如WMT机器翻译和CNN/DailyMail摘要任务上达到与领先非自回归基线相当的性能，同时相比自回归Transformer具有显著的速度优势。进一步证明，可学习的频域参数使模型能自适应聚焦于长程或短程依赖，部分缓解了一次性生成中的连贯性差距。总体而言，该工作展示了将频域操作融入序列生成的潜力，可为大模型推理任务带来显著的计算与时间节省。
原文摘要 · Abstract (English)
We present FourierNAT, a novel non-autoregressive Transformer (NAT) architecture that employs Fourier-based mixing in the decoder to generate output sequences in parallel. While traditional NAT approaches often face challenges with capturing global dependencies, our method leverages a discrete Fourier transform to mix token embeddings across the entire sequence dimension, coupled with learned frequency-domain gating. This allows the model to efficiently propagate context without explicit autoregressive steps. Empirically, FourierNAT achieves competitive results against leading NAT baselines on standard benchmarks like WMT machine translation and CNN/DailyMail summarization, providing significant speed advantages over autoregressive Transformers. We further demonstrate that learned frequency-domain parameters allow the model to adaptively focus on long-range or short-range dependencies, partially mitigating the well-known coherence gaps in one-pass NAT generation. Overall, FourierNAT highlights the potential of integrating spectral-domain operations to accelerate and improve parallel text generation. This approach can potentially provide great computational and time savings in inference tasks LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。