用分组替代掩码，实现高效并行生成。
Partition Generative Modeling: Masked Modeling Without Masks
- 将令牌分为互不交互的两组，通过条件预测生成
- 在OpenWebText上吞吐量提升5.5倍，困惑度更低
- 兼容现有采样器和蒸馏方法，适合大规模生成
掩码生成模型（MGMs）可并行、任意顺序生成标记，优于一次生成一个标记的自回归模型（ARMs）。但MGMs在每步采样中仍处理完整序列，包括无信息的掩码标记。而ARMs仅处理已生成标记。我们提出「分区生成模型」（PGMs），以分组代替掩码：将标记分为两个互不注意的组，模型学习基于另一组预测本组，彻底消除掩码标记。由于两组不交互，采样时只需处理干净标记，类似ARMs；同时保持MGMs的并行与任意顺序生成能力。在OpenWebText上，PGMs比MDLM快5-5.5倍，且生成困惑度更低；在ImageNet上，与MaskGIT达到相当的FID，吞吐提升7.5倍；使用双倍步数后FID达4.56，仍比MGMs快3.9倍。PGMs还兼容现有MGM采样器与蒸馏方法。
原文摘要 · Abstract (English)
Masked generative models (MGMs) can generate tokens in parallel and in any order, unlike autoregressive models (ARMs), which decode one token at a time, left-to-right. However, MGMs process the full-length sequence at every sampling step, including mask tokens that carry no information. In contrast, ARMs process only the previously generated tokens. We introduce ``Partition Generative Models'' (PGMs), which replace masking with partitioning. Tokens are split into two groups that cannot attend to each other, and the model learns to predict each group conditioned on the other, eliminating mask tokens entirely. Because the groups do not interact, PGMs can process only the clean tokens during sampling, like ARMs, while retaining parallel, any-order generation, like MGMs. On OpenWebText, PGMs achieve $5-5.5\times$ higher throughput than MDLM while producing samples with lower Generative Perplexity. On ImageNet, PGMs reach comparable FID to MaskGIT with a $7.5\times$ throughput improvement. With twice as many steps, the FID improves to 4.56 while remaining $3.9\times$ faster than MGMs. Finally, PGMs remain compatible with existing MGM samplers and distillation methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。