将上下文编码与去噪分离，实现更快的文本生成。
Nemotron-Labs-TwoTower: Diffusion Language Modeling with Pretrained Autoregressive Context
- 用两个独立模块分别处理上下文和去噪，提升效率
- 生成速度比基线快2.42倍，质量损失仅1.3%
- 适合需要高速生成的场景，如实时对话系统
扩散语言模型因其并行迭代生成的潜力，被视为自回归模型的有前景替代方案。然而，现有方法使用单一网络同时承担上下文表示与迭代去噪任务，导致模型在两项功能上均受限。我们提出TwoTower，一种分块自回归扩散模型，将两种角色解耦为两个塔：一个冻结的自回归上下文塔，用于因果处理干净标记；一个可训练的扩散去噪塔，采用双向块注意力，通过跨注意力机制对噪声块进行精炼。该模型基于Nemotron-3-Nano-30B-A3B（一个300亿参数的开源混合Mamba-Transformer MoE模型），在约2.1万亿标记数据上训练，保留了自回归基线98.7%的质量，同时实现2.42倍更高的墙钟生成吞吐量。代码与模型权重已公开于https://huggingface.co/collections/nvidia/nemotron-labs-twotower。
原文摘要 · Abstract (English)
Diffusion language models offer a promising alternative to autoregressive models due to their potential for parallel and iterative generation. However, existing approaches use a single network for both context representation and iterative denoising, forcing one model to serve both roles and limiting its capacity for either role. We propose TwoTower, a block-wise autoregressive diffusion model that decouples these roles into two towers: a frozen AR context tower that causally processes clean tokens, and a trainable diffusion denoiser tower with bidirectional block attention that refines noisy blocks via cross-attention to the context. Built on Nemotron-3-Nano-30B-A3B, an open-weight 30B hybrid Mamba-Transformer MoE model, and trained on approximately 2.1T tokens, Nemotron-Labs-TwoTower retains 98.7% of the autoregressive baseline's quality while offering 2.42X higher wall-clock generation throughput. We release the code and model weights at https://huggingface.co/collections/nvidia/nemotron-labs-twotower.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。