提出块生成框架BlockGen,实现更灵活的序列建模与采样优化。
BlockGen: Flexible Blockwise Sequence Modeling with Hybrid Samplers
- 采用混合块大小训练,灵活衔接自回归与纯扩散模型。
- 在16块大小下,均匀扩散在少步采样中优于掩码扩散。
- 引入自回归指导修正采样,无需额外验证器,适合高效生成任务。
统一状态扩散模型(USDM)是否比掩码扩散模型(MDM)更具优势?近期研究表明,结合预测-修正采样器时,USDM生成样本质量更高,在下游任务中表现相当或更优,尽管其困惑度更高。但现有研究存在两个未解问题:一是使用无信息的修正器随机重注噪声,而非针对最可能出错的词元;二是仅比较全序列扩散模型,未考察分块生成场景。为此,本文提出BlockGen,一种基于块的序列建模方法,可实例化为掩码与统一扩散模型。BlockGen在混合块尺寸上训练,其似然值可更精细地在自回归(AR)与纯扩散之间插值。该模型支持自回归指导的预测-修正采样(ARPC),结合自回归与扩散预测,直接重构低概率词元,无需辅助验证器。在祖先采样下,均匀扩散在分块设置中表现更优,尤其在少步数(NFE)情况下;在高NFE时,通过ARPC使差距缩小甚至反转。在GSM8K数据集上,块大小为16时,MDM略高于USDM的准确率,且在OpenWebText上的生成困惑度趋势一致。代码已开源。
原文摘要 · Abstract (English)
Is the uniform-state diffusion framework a more powerful paradigm for discrete diffusion? Recent studies indicate that this may be the case. In combination with predictor-corrector samplers, uniform-state diffusion models (USDMs) produce samples of higher-quality than masked diffusion models (MDMs), and USDMs equal or outperform MDMs in downstream tasks, even though they exhibit greater perplexity. Two issues remain unresolved. First, existing work compares uniform and masked diffusion with un-informed correctors that re-inject noise at random positions, rather than targeting tokens most likely to be wrong. Second, prior work compares full-sequence diffusion models, so we do not know whether the same conclusion holds when tokens are generated block by block. To address these issues, we introduce BlockGen, a blockwise sequence model that we instantiate with both masked and uniform diffusion. BlockGen trains on a mixture of block sizes and its likelihood interpolates between AR and pure diffusion more finely than models with a fixed block size. BlockGen enables AR-informed predictor-corrector sampling (ARPC), which combines AR and diffusion predictions to re-generate unlikely tokens without an auxiliary verifier. Under ancestral sampling, uniform outperforms masked in the block-by-block setting, especially in the few-step regime. Under ARPC, the gap closes and reverses at high NFE. With block size $16$ on GSM8K, MDMs reach slightly higher accuracy than USDMs, and we observe a similar trend in Generative Perplexity on OpenWebText. Find our code at https://github.com/jdeschena/blockgen.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。