提出TUBE方法，精准评估离散扩散语言模型的真实生成能力。
TUBE: Tangent Upper Bound on Evidence for Discrete Diffusion Language Models
- 基于切线思想构造可无偏估计的对数似然上界
- 实证发现扩散模型似然值严格低于自回归基线
- 适用于多种生成模型，尤其适合对比评估
对数似然值是生成模型的标准评价指标。然而，与自回归模型（ARMs）不同，离散扩散模型通常无法精确计算该值。现有评估依赖证据下界（ELBO），但无法判断真实值可能高出多少。本文提出切线上界（TUBE），一种可无偏蒙特卡洛估计的对数似然上界，适用于包含掩码扩散模型（MDMs）、任意顺序自回归模型（AO-ARMs）及其块结构变体在内的潜变量模型。在块结构MDMs和块结构AO-ARMs上的应用表明，这些模型的似然值严格低于精确自回归基线，证明自回归模型在对数似然上仍占优势。
原文摘要 · Abstract (English)
Log-likelihood is a standard metric for evaluating generative models. Unfortunately, in contrast to autoregressive models (ARMs), discrete diffusion models generally do not admit exact computation of this quantity. Existing evaluations, therefore, rely on the evidence lower bound (ELBO), leaving unclear how much higher the true value may be. We address this by introducing the Tangent Upper Bound on Evidence (TUBE), a variational upper bound on log-likelihood that admits an unbiased Monte Carlo estimator. Our TUBE extends across latent-variable models, including masked diffusion models (MDMs), any-order ARMs (AO-ARMs), and block variants of both. Applied to block MDMs and block AO-ARMs, TUBE reveals our key empirical finding that these models lie strictly below the exact ARM baseline, showing that ARMs still dominate in likelihood.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。