用8步生成长文本,速度比传统方法快128倍且质量相当。
FS-DFM: Fast and Accurate Long Text Generation with Few-Step Diffusion Language Models
- 通过显式控制采样步数,实现快速生成。
- 8步采样即达1024步基线的困惑度水平。
- 适合需要低延迟、高吞吐的长文本生成场景。
自回归语言模型(ARMs)虽具备强似然性,但生成过程串行,每轮仅产出一个词元,限制吞吐量并增加长序列生成延迟。扩散语言模型(DLMs)可并行处理位置,理论上更高效,但传统离散扩散通常需数百至数千次模型评估才能达到高质量,以迭代广度换串行深度。本文提出FS-DFM:少步离散流匹配模型。核心思想是将采样步数作为可调节参数,训练模型在不同步数预算下保持一致性,使一次大步移动等效于多次小步移动。结合可靠更新规则与从长期轨迹中蒸馏的教师指导,实现少步采样的稳定、准确与可控。在语言建模基准上,采用相似规模模型,FS-DFM仅用8步采样即可在生成1,024个词元时达到1,024步离散流基线的困惑度表现,采样速度提升最高达128倍,显著降低延迟并提升吞吐量。代码与预训练检查点见:https://github.com/apple/ml-fs-dfm
原文摘要 · Abstract (English)
Autoregressive language models (ARMs) deliver strong likelihoods, but are inherently serial: they generate one token per forward pass, which limits throughput and inflates latency for long sequences. Diffusion Language Models (DLMs) parallelize across positions and thus appear promising for language generation, yet standard discrete diffusion typically needs hundreds to thousands of model evaluations to reach high quality, trading serial depth for iterative breadth. We introduce FS-DFM, Few-Step Discrete Flow-Matching. A discrete flow-matching model designed for speed without sacrificing quality. The core idea is simple: make the number of sampling steps an explicit parameter and train the model to be consistent across step budgets, so one big move lands where many small moves would. We pair this with a reliable update rule that moves probability in the right direction without overshooting, and with strong teacher guidance distilled from long-run trajectories. Together, these choices make few-step sampling stable, accurate, and easy to control. On language modeling benchmarks, FS-DFM with 8 sampling steps achieves perplexity parity with a 1,024-step discrete-flow baseline for generating 1,024 tokens using a similar-size model, delivering up to 128 times faster sampling and corresponding latency/throughput gains. Code & pretrained checkpoints: https://github.com/apple/ml-fs-dfm
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。