用编码器-解码器结构加速离散扩散语言模型的生成与训练。
Encoder-Decoder Diffusion Language Models for Efficient Training and Inference
- 分离清洁词元表示与去噪过程，编码器专注表征，解码器轻量迭代优化。
- 在摘要、翻译和数学推理任务上，生成质量高且推理吞吐提升显著。
- 适合追求高速生成与高质量输出的自然语言生成应用者。
离散扩散模型可并行采样以实现比自回归方法更快的推理速度。然而，以往的扩散模型采用仅解码器架构，在每个去噪步骤中需调用完整网络，导致计算成本高。本文关键洞察是：离散扩散模型包含两类计算——1）表示干净词元；2）对污染词元进行去噪，这使得为每类任务使用独立模块成为可能。为此，我们提出一种编码器-解码器架构，通过编码器表示干净词元，由轻量解码器逐步重构噪声序列，从而加速推理。同时，该架构也支持块扩散模型的快速训练，后者将序列分块以提升生成质量，常用于扩散语言模型推理。我们构建了高效编码器-解码器扩散框架（E2D2），包含专用训练与采样算法，实验表明其在摘要、翻译和数学推理任务上实现了生成质量与推理吞吐之间的更优权衡。代码、模型权重及博客已发布于项目页面：https://m-arriola.com/e2d2。
原文摘要 · Abstract (English)
Discrete diffusion models enable parallel token sampling for faster inference than autoregressive approaches. However, prior diffusion models use a decoder-only architecture, which requires sampling algorithms that invoke the full network at every denoising step and incur high computational cost. Our key insight is that discrete diffusion models perform two types of computation: 1) representing clean tokens and 2) denoising corrupted tokens, which enables us to use separate modules for each task. We propose an encoder-decoder architecture to accelerate discrete diffusion inference, which relies on an encoder to represent clean tokens and a lightweight decoder to iteratively refine a noised sequence. We also show that this architecture enables faster training of block diffusion models, which partition sequences into blocks for better quality and are commonly used in diffusion language model inference. We introduce a framework for Efficient Encoder-Decoder Diffusion (E2D2), consisting of an architecture with specialized training and sampling algorithms, and we show that E2D2 achieves superior trade-offs between generation quality and inference throughput on summarization, translation, and mathematical reasoning tasks. We provide the code, model weights, and blog post on the project page: https://m-arriola.com/e2d2
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。