arXiv:2603.01367cs.LG2026-03被引 6

首次实现掩码扩散模型在测试分布下的精确似然计算,让其困惑度评估真正可信。

DUEL: Exact Likelihood for Masked Diffusion via Deterministic Unmasking

  • 提出DUEL框架,通过确定性解码位置策略实现测试时分布下的精确似然。
  • 在同域数据上困惑度差距缩小32%,零样本任务上缩小82%,性能远超此前估计。
  • 可公平比较不同采样方法,揭示掩码扩散模型潜力远未达上限。

掩码扩散模型(MDMs)通过迭代选择位置解码并预测对应词元生成文本,但缺乏准确的似然评估:证据下界(ELBO)不仅对对数似然的下界估计松散,且在训练分布而非测试分布下计算。本文提出的DUEL框架统一了采用确定性位置选择的主流采样策略,证明其在测试时分布下可实现精确似然计算,首次为MDMs赋予正确的似然与困惑度。该正确困惑度是自回归模型困惑度的自然类比,使我们能重新审视关键问题:MDMs实际表现远优于以往认知——在同域数据上困惑度差距缩小32%,零样本任务上缩小82%。DUEL支持跨计算预算的高效采样器原理化比较,揭示一种强默认方法。此外,最优位置顺序搜索显示MDMs可超越自回归模型,于AG News上达到36.47对比52.11的困惑度,表明其性能上限尚未触及。

原文摘要 · Abstract (English)

Masked diffusion models (MDMs) generate text by iteratively selecting positions to unmask and then predicting tokens at those positions. Yet MDMs lack proper likelihood evaluation: the evidence lower bound (ELBO) is not only a loose bound on log-likelihood, but, as we show, is also computed under the training distribution rather than the test-time distribution. We resolve this within our DUEL framework, which unifies leading MDM sampling strategies that employ $\textit{deterministic}$ position selection. We prove that DUEL samplers admit $\textbf{exact likelihood computation under the test-time distribution}$ -- giving MDMs $\textit{proper}$ likelihood, and hence proper perplexity, for the first time. This proper perplexity is the natural analogue of autoregressive perplexity and lets us revisit key questions about MDMs. $\textbf{MDMs are substantially better than previously thought}$: the MDM-autoregressive perplexity gap shrinks by up to $32\%$ on in-domain data and $82\%$ on zero-shot benchmarks. DUEL enables the first principled comparison of fast,parallel samplers across compute budgets -- an analysis impossible with the ELBO and unreliable with generative perplexity -- identifying a strong default method. Finally, oracle search over position orderings reveals MDMs can far surpass autoregressive models -- achieving $36.47$ vs. $52.11$ perplexity on AG News -- demonstrating the ceiling of MDM performance has not yet been reached.

扩散模型文本生成似然计算困惑度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。