用新方法让并行生成模型更像自回归模型，提升文本质量。
Gumbel Distillation for Parallel Text Generation
- 通过高斯-最大技巧将教师模型的噪声空间映射为输出序列
- 在OpenWebText上使并行模型的生成困惑度降低10.5%
- 适用于多种并行架构，适合追求高效高质量生成的研究者
自回归语言模型因逐词生成而速度慢，促使人们采用并行解码方法。然而这些非自回归模型常因难以建模词元序列的复杂联合分布而导致生成质量下降。为此，我们提出一种名为Gumbel Distillation的新蒸馏技术，使并行解码器能有效学习该分布。该方法利用Gumbel-Max技巧，从高性能自回归教师模型的潜在高斯噪声空间构建确定性输出映射。作为模型无关技术，Gumbel Distillation可无缝集成至MDLM和BD3-LM等多种并行解码架构。在LM1B和OpenWebText数据集上的实验表明，该方法显著提升了并行语言模型的生成质量：在OpenWebText上，相比未使用蒸馏的MDLM，MAUVE得分提升30.0%，生成困惑度降低10.5%。代码已公开于https://github.com/hxixixh/gumbel-distill。
原文摘要 · Abstract (English)
The slow, sequential nature of autoregressive (AR) language models has driven the adoption of parallel decoding methods. However, these non-AR models often sacrifice generation quality as they struggle to model the complex joint distribution of token sequences. To narrow this performance gap, we introduce Gumbel Distillation, a novel distillation technique that enables parallel decoders to learn this distribution effectively. Our method leverages the Gumbel-Max trick to create a deterministic mapping from a latent Gumbel noise space to the output tokens of a high-performing AR teacher. As a model-agnostic technique, Gumbel Distillation seamlessly integrates with diverse parallel decoding architectures, including MDLM and BD3-LM. Experiments on LM1B and OpenWebText show that Gumbel Distillation substantially improves the generation quality of parallel language models, achieving a 30.0% improvement in MAUVE score and 10.5% in generative perplexity over MDLM trained on OpenWebText dataset. Code available at https://github.com/hxixixh/gumbel-distill.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。